cutting big data down to size with amd and dell

COST-EFFECTIVE SOLUTIONS FOR BIG DATA WITH HADOOP

Matt Kimball Product Marketing Manager Dell World 2012 Austin, Texas December 13, 2012

2 | Cutting Big Data Down to Size | Dell World – December 2012 |

WHAT‘S BIG DATA?

BIG DATA

Data sets sized beyond the ability of commonly used tools to capture, manage, and process the

data within a tolerable time.

Definitions derived from Gartner, IDC, and NIST documents

BIG DATA Technologies

A new generation of technologies designed to economically extract value from very large

volumes of a wide variety of data.


DROWNING IN DATA?

2.5 quintillion

bytes of data created

daily*

Growing numbers of

volumes

Increasing data complexity

Proliferation of unstructured

data

Surge in external data

streams


WHAT‘S HADOOP?

• Scalable, fault-tolerant, distributed system for data storage and processing

• Foundation of rapidly growing ecosystem of supporting projects

• Created by developers to handle intractable problem (Cloudera, Ebay, Facebook, Hortonworks, Linkedin, Twitter, and Yahoo!)

• Open source project hosted under the Apache Software Foundation

http://hadoop.apache.org/

Framework for distributed processing of large data sets across clusters of computers using a simple programming model

http://hadoop.apache.org/

http://hadoop.apache.org/common/

http://hadoop.apache.org/mapreduce/

http://hadoop.apache.org/hdfs/


HOW IS HADOOP BEING USED? A

DV

AN

CE

D

AN

AL

YT

ICS

DA

TA

PR

OC

ES

SIN

G

Web

Media

Telco

Retail

Financial

Federal

Social network analysis

Content optimization

Network analytics

Loyalty and promotions analysis

Fraud analysis

Entity analysis

Clickstream sessionization

Clickstream sessionization

Mediation

Data factory

Trade reconciliation

SIGINT

INDUSTRY TERM INDUSTRY TERM VERTICAL

Source: Cloudera


WHO IS USING, CONTRIBUTING, AND BUILDING FOR HADOOP?

10Gen Concurrent Klout SnapLogic

Accela Datameer LexisNexis StackIQ

Adobe eBay LinkedIn StumbleUpon

Alibaba.com eHarmony Microsoft SunGard

Amazon.com EMC NetApp Talend

AOL EnterpriseDB Netflix Telenav

AT&T Google Nokia Teradata

Baidu Groupon NTT Communications The New York Times

Bank of America Hortonworks Orbitz Trend Micro

bitly Hulu Pandora Twitter

Booz Allen Hamilton IBM Pervasive Tynt

CBS IIIT Hyderabad Rackspace Hosting Vertica

CBS Interactive Intel Samsung Visa

Cisco Jaspersoft SGI VoltDB

CNET JPMorgan Chase Sky Yahoo

Source: Hortonworks And of course AMD!

http://wiki.apache.org/hadoop/PoweredBy




WHY WAS HADOOP CREATED?

Exploding data volumes and types

Dramatic changes in enterprise data management

• It’s difficult to handle data this diverse at this scale • Traditional platforms can’t keep pace

• Any type any kind

• From any source

• Structured and unstructured

• At scale

• Deep analysis

• Exhaustive and detailed

• Sophisticated algorithms

• Quick results

HARD PROBLEMS

BIG DATA

• Extract more value from

more data more cost

effectively with greater

flexibility

NEW

OPPORTUNITY

FOR USING

CREATED DATA

WEB

LOGS SOCIAL

MEDIA

TRANSACTIONAL

DATA

SMART

GRIDS

OPERATIONAL

DATA

DIGITAL

CONTENT

R&D

DATA

AD

IMPRESSIONS

FILES

LEADS TO

Source: Cloudera


TRENDS AROUND BIG DATA

Big Data Trends:

• Low acquisition costs

• Servers that can effectively parallelize workloads

• Cost effectively scale out Hadoop systems to meet expanding data processing and analysis needs

• Compute density to handle large amounts of data

Enterprise and Government organizations faced with analyzing large amounts of unstructured data.

Data Driven Business Decisions and Actions


AMD AND DELL FOR YOUR HADOOP CLUSTERS

• Medium clock speeds CPU

• Two sockets Form Factor

• Concern as cluster grows Power

• 48GB to keep CPUs busy RAM

• 1 spindle per core – SATA Storage

• 1GB growing to InfiniBand™ Network

Three P Equilibrium

Core Density for Parallelism

Scale Out Capability

Manageability

Cost Per Node


TCO

Perf

Had

oo

p D

eplo

ym

ent

Scen

ario

2

4 T

B D

ata

10

Use

rs

I/O

Bo

un

d

CapEx

OpEx

Core Count

Scale Out

C6105 AMD 4274HE

64GB RAM

AMD OPTERON™ SERIES PROCESSORS FOR HADOOP LOWEST TCO FOR DELL CUSTOMERS


AMD HADOOP SCENARIO TCO

OpEx

CapEx

* See slide titled ” Substantiation | Hadoop TCO Calculations” for assumptions and calculations

Up to 34% Lower Hardware Costs11

Up to 40% Lower* CapEx

Dell C6105

HP SL230s Gen8

$0

$200,000

$400,000

$600,000

$800,000

Server HW cost Server SW cost Rack switch cost Rack + install cost CapEx

$401,500

$18,342 $3,000

$422,842

$605,120

$0 $87,920

$12,000

$705,040

Up to 23% Lower OpEx*

Up to 32% lower 3 year TCO - $407,329 savings*

Dell C6105

HP DL360e Gen8

$0

$50,000

$100,000

$150,000

$200,000

Server powercost/year

Overheadpower cost/year

Mgmt OH/year Footprint cost OpEx/yr

$15,773 $12,619 $8,680

$142,267

$24,479 $19,583

$105,195

$34,720

$183,977


80 Servers 1 42U rack (C6105) vs. 4 42U racks (DL360e Gen8)

Mellanox SX2016 10GBe switches: 2 for C6105 vs. 8 for DL360e Gen8

Hadoop Projects Software

HADOOP CONFIGURATION COMPARISON DELL POWEREDGE C6105 VS HP DL360E GEN8

HP DL360e Gen8 CPU Intel Xeon E5-2440 (2)

Memory 64 GB (8x8 1333MHz)

Storage 3 x 2TB LFF SATA

Network 1GBe

PSU 750W

Dell™ PowerEdge™ C6105 (4 nodes)

CPU / Node AMD Opteron™ 4274HE (2)

Memory / Node 64 GB (8x8 1333MHz)

Storage / Node 3x 2TB SATA

Network / Node 1GBe

PSU / Node 1100W


EUROPEAN AEROSPACE PROJECT HADOOP SYSTEM

• 4 service nodes • Cluster infrastructure – 5 Nodes • 94 x C6105 – 742 Sockets • 4x MB w/ 2x 4274HE, 24GB, 3x 3TB • Dell|Force10 + Dell switches

2013 Production

Astronomy Mission Mapping Our Galaxy


DELL CLOUDERA SOLUTIONS

http://www.mellanox.com/webinars/Accelerating-Big-Data-Over-Hadoop-With-InfiniBand/

The Dell | Hadoop Big Data Solution • Cloudera’s Distribution for Hadoop • Dell PowerEdge™ C hardware, including the C6105 and C6145

http://dell.cloudera.com/














http://www.dell.com/poweredgec

http://dell.cloudera.com/


THANK YOU

For more information on AMD Big Data solutions, please visit;

www.AMD.com/BigData


DISCLAIMER The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes.

AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY DIRECT, INDIRECT, SPECIAL OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

ATTRIBUTION © 2012 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, Radeon, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other names and logos are used for informational purposes only and may be trademarks of their respective owners.


BACK-UP


REFERENCE CONFIGURATION

Server: Dell™ PowerEdge™ C6105 (4-nodes)

CPU/node: AMD Opteron™ 4276HE (4)

Memory/node: 64 GB (8x8 1333MHz)

Storage/node: 3 x 2TB SATA

Network/node: 1 GBe

PSU/node: 1100W

Software:

OS: CentOS 5.x (or Red Hat)

Hadoop Distribution: Cloudera

JAVA: Sun Oracle JVM 1.6u2x

Monitoring: Nagios, Ganglia

Other: Hadoop Projects Software

Coming soon: AMD Opteron™ 4300 series processor support


AMD’S PILOT HADOOP ARCHITECTURE


HADOOP TCO CALCULATIONS


HADOOP FOR TELCO

Hadoop on C6105

4 x Motherboard system Each Motherboard: • 2 x4274HE Proc. • 4 x 16GB RAM • 4 x 2TB HDD


SUBSTANTIATION AND FOOTNOTES

1. SVR-217 - AMD Opteron™ 6300 Series processors are expected to have up to 37% higher java performance/watt than AMD Opteron 6200 Series processors.. Estimate based on preliminary measurements of java performance/watt in AMD labs as of August 2012 comparing AMD Opteron™ processor Models 6380 and 6278. SVR-217

2. SVR-168 - AMD Opteron™ 6300 Series processors are expected to have up to 24% higher Java performance than AMD Opteron 6200 Series processors. Estimate based on preliminary measurements of server side Java performance in AMD labs as of August 30, 2012. 1,199,838 operations per second using 2 x AMD Opteron™ processors Model 6278. 1,489,668 operations per second using 2 x AMD Opteron™ processors Model 6380. SVR-168

3. SVR-205 - AMD Opteron™ 6300 processors are expected to offer up to 8% better integer performance than AMD Opteron™ 6200 processors. Estimate based on preliminary SPECint®_rate2006 results in AMD labs as of August 2012 comparing AMD Opteron™ processor Models 6380 and 6278.

4. SVR-216 - AMD Opteron™ 6300 processors are expected to offer up to 7% better floating point performance than AMD Opteron™ 6200 processors. Estimates based on preliminary SPECfp®_rate2006 measurements in AMD labs as of August 2012 comparing AMD Opteron™ processor Models 6380 and 6278 . SVR-216

5. SVR-255 - The Dell R815 lowers TCO for virtualization by up to 17% versus the HP DL560 Gen8. TCO calculation based on virtualization scenario with 1600 physical servers consolidated down to 40 with each physical server running 40 virtualized machines (12GB RAM each). This is supported by two racks of (20) servers with two Mellanox SX1016 10GBe 64-port switches. The cost per rack with installation is $3,000 based on http://www.intel.com/technology/eep/datacenter.pdf as of 9/10/12 and (2) Mellanox SX1016 10GBe 64-port switches cost $10,990 each and have a typical power consumption of 113 watts as of 9/10/12 at http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1377. Power costs based on $0.10 per kWhr and using 976 watts per for the Dell R815 server and 588 watts for the HP DL560p Gen8 server according to the Dell Energy Smart Solution Advisor (ESSA) tool with a memory-intensive workload set at 100% CPU loading and an input voltage of 220 volts as well as the HP Power Advisor Tool set at 100% CPU loading and an input voltage of 220 volts as of 9/24/12 based on the configurations below. The PUE is 1.8. Assuming 28 square feet per rack according to http://www.energystar.gov/ia/partners/prod_development/downloads/Additional_FAQs_LightingCooling_Racks.pdf as of 9/11/12, and the average fully burdened cost of real estate at $310 per square foot per year based on http://www.vmware.com/technology/whyvmware/calculator/index.php as of 9/10/12. Management overhead yearly cost per server based on $81k salary with 1.25 multiplier for a fully loaded expense (benefits, taxes, etc) of $101,250. $1315 per server derived from dividing the fully loaded expense by 77 servers being managed by IT administrator, based on page 4 at http://www.vmware.com/files/pdf/vmware-solution-opex-reducing-opex-wp-en.pdf and http://www.indeed.com/salary/Windows-Server-Administrator.html. VMware vSphere license is $3495 per CPU as of 9/25/12 at www.vmware.com. R815 with four AMD Opteron™ processor Model 6274, 512GB RAM, 2 x 1TB SATA drives, two 10GBe NICs, and 3yr base warranty cost $20140 as of 9/24/12 at www.dell.com. HP DL560p Gen8 with four Intel Xeon processor Model E5-4620, 512GB RAM, 2 x 1TB SATA drives, two 10GBe NICs, and 3yr base warranty cost $30948 as of 9/24/12 at www.hp.com. The 3yr TCO using AMD Opteron™-based servers is $1,811,446. The 3yr TCO using Intel Xeon-based server is $2,170,350. Assuming customer already has agreement for client license and run time license, so not including Windows Server license. Actual results will vary. SVR-255


SUBSTANTIATION AND FOOTNOTES, CONTINUED

6. SVR-254 - The Dell R815 offers up to a 35% lower server hardware acquisition cost than the HP DL560 Gen8 for virtualization configurations. R815 with four AMD Opteron™ processor Model 6274, 512GB RAM, 2 x 1TB SATA drives, two 10GBe NICs, and 3yr base warranty cost $20140 as of 9/24/12 at www.dell.com. HP DL560 Gen8 with four Intel Xeon processor Model E5-4620, 512GB RAM, 2 x 1TB SATA drives, two 10GBe NICs, and 3yr base warranty cost $30948 as of 9/24/12 at www.hp.com. SVR-254

7. SVR-213 - Virtualized AMD-based servers deliver up to 12% higher performance and up to 23% better price performance than the competition. Testing with VMware ESX in today’s typical virtualized environments consisting of mid-range processor and memory configurations, reveals that AMD Opteron processors lead in both price as well as price/performance versus Intel-based systems. Virtualized AMD-based servers deliver up to 12% higher performance and up to 23% better price performance than the competition. Test run VMware ESX with DVD Store, an open source performance tool that includes web and database servers to emulate an ecommerce environment.. Servers running increasing number of VMs from 2 to 32 with a fixed, utilization rate of ~25% . Systems include: 2X AMD Opteron™ processors Model 6274 in HP DL385 Gen8 server running 4,233 Operations per Minute (OPM) for 2 VMs to 1,242 OPM for 32 VMs with 96GB (12x8 GB DDR3-1333) memory, 1x15K 500GB SAS HD, base warranty and costing $7076 for the hardware and $6990 for VMware vSphere Enterprise Plus software; 2 x Intel Xeon E5-2665 processors in a HP DL380p Gen 8 server running 3,769 OPM for 2 VMs to 1,178 OPM for 32 VMs, 96GB (12x8 GB DDR3-1333) memory, 1x15K 500GB SAS HD, base warranty and costing $9,338 for hardware and $6990 for VMware vSphere Enterprise Plus software . System prices as of 7/5/2012 http://www.hp.com and software as of 7/5/2012 http://vmware.com. SVR-213

8. SVR-303 - The Dell C6145 lowers TCO for VDI by up to 28% versus the HP SL230s Gen8. For the Dell C6145, the TCO calculation is based on a VDI scenario with 8000 users, 200 VDI sessions per server, and 2.5GB physical RAM per session, which is supported by one rack of (20) two-node 4P servers and two Mellanox SX1016 10GBe 64-port switches. For the HP SL230s Gen8, the TCO calculation is based on a VDI scenario with 8000 users, 100 VDI sessions per server, and 2.5GB physical RAM per session, which is supported by one rack of (80) 2P servers and two Mellanox SX1016 10GBe 64-port switches. The cost per rack with installation is $3,000 based on http://www.intel.com/technology/eep/datacenter.pdf as of 9/10/12 and (2) Mellanox SX1016 10GBe 64-port switches cost $10,990 each and have a typical power consumption of 113 watts as of 9/10/12 at http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1377. Power costs based on $0.10 per kWhr and using 1650 watts per for the Dell C6145 server and 417 watts for the HP SL230s server plus 182 watts for the C6500 enclosure according to the Dell Energy Smart Solution Advisor (ESSA) tool with a memory-intensive workload set at 100% CPU loading and an input voltage of 220 volts as well as the HP Power Advisor Tool set at 100% CPU loading and an input voltage of 220 volts as of 9/26/12 based on the configurations below. The PUE is 1.8. Assuming 28 square feet per rack according to http://www.energystar.gov/ia/partners/prod_development/downloads/Additional_FAQs_LightingCooling_Racks.pdf as of 9/11/12, and the average fully burdened cost of real estate at $310 per square foot per year based on http://www.vmware.com/technology/whyvmware/calculator/index.php as of 9/10/12. Management overhead yearly cost per server based on $81k salary with 1.25 multiplier for a fully loaded expense (benefits, taxes, etc) of $101,250. $1315 per server derived from dividing the fully loaded expense by 77 servers being managed by IT administrator, based on page 4 at http://www.vmware.com/files/pdf/vmware-solution-opex-reducing-opex-wp-en.pdf and http://www.indeed.com/salary/Windows-Server-Administrator.html. XenDesktop VDI software costs $134.75 per VDI session as of 9/10/12 provided by Sales at Insight. Dell C6145 with two server nodes of four AMD Opteron™ processor Model 6274, 512GB (32x16 1600MHz) RAM, 2 x 1TB SATA drives, one 10GBe NICs, and 3yr 24x7 warranty cost $43978 as of 10/4/12 according to Dell Sales. HP SL230s Gen8 with two Intel Xeon processor Model E5-2665, 256GB (16x16 1600MHz) RAM, 2 x 1TB SATA drives, one 10GBe NICs, and 3yr 24x7 warranty cost $19669 plus $2008 for the HP C6500 enclosure as of 10/4/12 at www.hp.com. The 3yr TCO using AMD Opteron™-based servers is $2,323,545. The 3yr TCO using Intel Xeon-based servers is $3,205,690. Assuming customer already has agreement for client license and run time license, so not including Windows Server license. Actual results will vary. SVR-303

http://www.hp.com/


SUBSTANTIATION AND FOOTNOTES, CONTINUED

9. SVR-302 - The Dell C6145 offers up to a 44% lower server hardware acquisition cost than the HP SL230s Gen8 for VDI configurations. Dell C6145 with two server nodes of four AMD Opteron™ processor Model 6274, 512GB (32x16 1600MHz) RAM, 2 x 1TB SATA drives, one 10GBe NICs, and 3yr 24x7 warranty cost $43978 as of 10/4/12 according to Dell Sales. HP SL230s Gen8 with two Intel Xeon processor Model E5-2665, 256GB (16x16 1600MHz) RAM, 2 x 1TB SATA drives, one 10GBe NICs, and 3yr 24x7 warranty cost $19669 as of 10/4/12 at www.hp.com. SVR-302

10. SVR-301 - The Dell C6105 lowers TCO for Hadoop by 32% versus the HP DL360e Gen8. TCO calculation based on Hadoop scenario with 24TB dataset being analyzed by 10 users and being processed by an I/O bound application. This is supported by one rack of 79 Dell C6105 slave nodes and 1 Dell C6105 master node along with two Dell Force10 S60 High-Performance 1/10 GbE switches as well as four racks of 79 HP DL360e Gen8 slave nodes and 1 HP DL360e Gen8 master node along with two Mellanox SX1016 10GBe 64-port switches per rack. The cost per rack with installation is $3,000 based on http://www.intel.com/technology/eep/datacenter.pdf as of 9/10/12. Mellanox SX1016 10GBe 64-port switches cost $10,990 each and have a typical power consumption of 113 watts as of 9/10/12 at http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1377. Dell Force10 S60 High-Performance 1/10 GbE switches cost $9171 each and have a typical power consumption of 156 watts as of 9/26/12 at www.dell.com and http://www.networkcomputing.com/next-gen-network-tech-center/force10s-deep-buffer-switch/229501471. Power costs based on $0.10 per kWhr and using 889 watts per for the Dell C6105 with four server nodes in one system and 338 watts for the HP DL360e Gen8 server with a transactional workload set at 100% CPU loading according to the Dell Energy Smart Solution Advisor (ESSA) tool with a transactional workload set at 100% CPU loading and an input voltage of 220 volts and the HP Power Advisor tool set at 100% CPU loading and an input voltage of 220 volts as of 9/26/12 based on the configurations below. The PUE is 1.8. Assuming 28 square feet per rack according to http://www.energystar.gov/ia/partners/prod_development/downloads/Additional_FAQs_LightingCooling_Racks.pdf as of 9/11/12, and the average fully burdened cost of real estate at $310 per square foot per year based on http://www.vmware.com/technology/whyvmware/calculator/index.php as of 9/10/12. Management overhead yearly cost per server based on $81k salary with 1.25 multiplier for a fully loaded expense (benefits, taxes, etc) of $101,250. $1315 per server node derived from dividing the fully loaded expense by 77 servers being managed by IT administrator, based on page 4 at http://www.vmware.com/files/pdf/vmware-solution-opex-reducing-opex-wp-en.pdf and http://www.indeed.com/salary/Windows-Server-Administrator.html. Open Source software is being used so there is no cost. Each of the four Dell C6105 server nodes has two AMD Opteron™ processor Model 4274HE, 64GB RAM, 3 x 2TB SATA drives, 1GBe, and 3yr base warranty with a total system cost of $20075 ($5019 each node) as of 10/3/12 from Dell Sales. HP DL360e with two Intel Xeon processor Model E5-2440, 64GB RAM, 3 x 2TB SATA drives, 1GBe, and 3yr base warranty cost $7564 as of 9/21/12 at www.hp.com. The 3yr TCO using AMD Opteron™-based servers is $849,643. The 3yr TCO using Intel Xeon-based servers is $1,256,972. Actual results will vary. SVR-301.

11. SVR-300 - The Dell C6105 offers up to a 34% lower scale-out server hardware acquistion cost than the HP DL360e Gen8 for Hadoop configurations. Each of the four Dell C6105 server nodes has two AMD Opteron™ processor Model 4274HE, 64GB RAM, 3 x 2TB SATA drives, 1GBe, and 3yr base warranty with a total system cost of $20075 ($5019 each node) as of 10/3/12 from Dell Sales. HP DL360e with two Intel Xeon processor Model E5-2440, 64GB RAM, 3 x 2TB SATA drives, 1GBe, and 3yr base warranty cost $7564 as of 9/21/12 at www.hp.com. SVR-300

cutting big data down to size with amd and dell

Documents

data processing

data storage

big data trends

whats big data

kindbig data

size dell world

wide variety of data

data volumes dramatic