cloudera-intel-cisco hadoop benchmark toi (external) what

Download Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What

Post on 07-Jan-2017

215 views

Category:

Documents

3 download

Embed Size (px)

TRANSCRIPT

  • Cloudera-Intel-Cisco Hadoop Benchmark TOI (External)What matters in a Hadoop Cluster?

    Floris Grandvarlet (Cisco) floris.grandvarlet@cisco.com

    Patrick Schotts (Intel) patrick.schots@intel.com

    Woody Christy (Cloudera) wchristy@cloudera.com

    mailto:floris.grandvarlet%40cisco.com?subject=mailto:patrick.schots%40intel.com?subject=mailto:wchristy%40cloudera.com?subject=

  • Cloudera-Intel-Cisco v2.0 Public Page 2

    AcknowledgmentsThe authors acknowledge the contributions of: Intel:Stephen G. Anderson, stephen.g.anderson@intel.comRob Kypriotakis, rob.kypriotakis@intel.comJacob A. Ohara, jacob.a.ohara@intel.comGert Pauwels, Gert.Pauwels@intel.comRichard B. Pilling, richard.b.pilling@intel.com

    Cisco:Arnaud Bassaler, abassale@cisco.comPeter Ruttens, pruttens@cisco.comMichel Sumbul, msumbul@cisco.comKarthik Kulkarni, kkulkar@cisco.com

    Cloudera:Sandeep Brahmarouthu, sandeep@cloudera.comJonathan Cooper, jcooper@cloudera.comRob Johnson, rj@cloudera.comKunal Kusoorkar, kkusoorkar@cloudera.comDwai Lahiri, dlahiri@cloudera.comJonathan Seidman, jseidman@cloudera.com

    ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, DESIGNS) IN THIS PAPER ARE PRESENTED AS IS, WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

    THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

    CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

    All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

    2014 Cisco Systems, Inc. All rights reserved.

    Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?

    mailto:stephen.g.anderson%40intel.com?subject=mailto:rob.kypriotakis%40intel.com?subject=mailto:jacob.a.ohara%40intel.com?subject=mailto:Gert.Pauwels%40intel.com?subject=mailto:richard.b.pilling%40intel.com?subject=mailto:abassale%40cisco.com?subject=mailto:pruttens%40cisco.com?subject=mailto:msumbul%40cisco.com?subject=mailto:kkulkar%40cisco.com?subject=mailto:sandeep%40cloudera.com?subject=mailto:jcooper%40cloudera.com?subject=mailto:rj%40cloudera.com?subject=mailto:kkusoorkar%40cloudera.com?subject=mailto:dlahiri%40cloudera.com?subject=mailto:jseidman%40cloudera.com?subject=

  • Cloudera-Intel-Cisco v2.0 Public Page 3

    Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?

    Contents1. Introduction ................................................................................................................................................................... 4

    Executive Summary ........................................................................................................................................................... 5

    2. Benchmark Test bed ...................................................................................................................................................... 72.1. Hardware ......................................................................................................................................................................... 72.2. Software .......................................................................................................................................................................... 72.3. Software post-installation configuration ........................................................................................................................... 82.4. Architecture ..................................................................................................................................................................... 92.5. Server Configuration and Cabling .................................................................................................................................... 102.6. Rack ................................................................................................................................................................................. 11

    3. CPU Benchmark ............................................................................................................................................................ 123.1. Overview .......................................................................................................................................................................... 123.2. CPU Test Architecture ..................................................................................................................................................... 123.3. CPU Benchmarks Caveats ............................................................................................................................................... 13

    3.3.1. Cloudera Manager Architecture ............................................................................................................................ 153.3.2. Power measurements ........................................................................................................................................... 16

    3.4. Results ............................................................................................................................................................................. 183.4.1. Tera Results for CPU ............................................................................................................................................. 183.4.2. Word Count for CPU ............................................................................................................................................. 193.4.3. Power Results for CPU .......................................................................................................................................... 203.4.4. Consolidated Results with Pricing ......................................................................................................................... 20

    3.5. CPU Benchmark Results Conclusion ............................................................................................................................... 21

    4. Cluster Benchmark ........................................................................................................................................................ 234.1. Overview .......................................................................................................................................................................... 234.2. Benchmark Caveat........................................................................................................................................................... 23

    4.2.1. Benchmark Caveat : Raid Configuration ................................................................................................................ 234.2.2. Benchmark Caveat : Network Bandwidth .............................................................................................................. 24

    4.3. Benchmark Hyper-Threading ........................................................................................................................................... 264.3.1. Hyper-Threading details ........................................................................................................................................ 27

    4.4. Benchmark Network Bandwidth ....................................................................................................................................... 284.4.1. TeraGen and TeraSort details ................................................................................................................................ 29

    4.5. Benchmark Hyper-Threading/Networking results conclusion .............................

Recommended

View more >