Shifting a Global Applications Services Business to Public CloudSpeaker: Paul Rafferty, Director of Infrastructure Architecture, Silverpop

Silverpop

• Cloud-based global marketing automation services business

• 2500 customers; 500+ employees; 2 managing storage

• Multitenant SaaS product -- 100s of customers per deployment

• Cutting edge big data analytics to analyze customer behavior

• Pre SoftLayer: 4 data centers; After SoftLayer: 10 data centers

1

Video: Silverpop Spectrum Accelerate Use Casehttps://youtu.be/y2uumq9IAZ0

Our cloud challenge

• Goal: Strategically shift from traditional data center to public cloud

storage to grow globally faster, be compliant, and save costs

• Challenges:

– Swiftly migrate mission critical customer data out of physical data centers

– Locate our services closer to our global customer base

– Meet new global regulations for privacy, data protection and transfer

– Meet our data protection and performance needs in a cloud model

• Solution:

– SoftLayer + IBM Spectrum Accelerate software defined storage (with

Supermicro as the underlying server hardware)2

Why we chose Spectrum Accelerate (1/4)

Data protection and performance

• We are paranoid about data protection – SaaS application hosting OUR

customers’ data – close to 1PB in total

• Typically maintain three online copies of all data on disparate arrays

• Primary production copy

• Secondary production copy

• Primary disaster recover copy

• All three must be able to support production workloads

• Very high IOP needs per system (25-75k IOPs)

• Very high throughput per system (1GB/s)

3

Why we chose IBM Spectrum Accelerate (2/4)

• over VMware Virtual SAN™ (VSAN)

– Enterprise management features

– Scale – need 100s of TBs per location

– Rich support matrix for diverse workloads (hosts,

clusters, files systems), not just VMware

• over EMC ScaleIO®

– Simpler, lower cost client model of iSCSI versus

driver: no need to buy FC HBAs

– XIV foundation is a more natural fit for

transitioning from on-premises arrays

– No replication and no data migration tools

– We do run about 30TB of ScaleIO in our on-

premises data centers

4

• over Open Source

– Maturity, scalability, proven reliability

– Quick to launch, low barrier to entry for SAN

admin

– For scaling, ease of use, predictable

performance, and features

• over SoftLayer storage

– Management of 1000s of LUNs is difficult

– Aggregate performance of an XIV array

versus 1000s of individual SoftLayer devices

– lower cost!

– LUN growth and migration

– Performance monitoring

Lots of options out there…

Why we chose Spectrum Accelerate (3/4)

It fits our workloads...

• We are very write heavy

– Workloads with over 20k+ random 8kb overwrites per second per database server

– XIV (and enterprise arrays) are very good at write workloads compared to mid-tier

solutions

– Data consistency is very important – XIV lineage has years of dealing with the

vagaries of SATA drives

• We have large databases – support and manage 100s of LUNs.

– Smallest Oracle databases are 20TB with an average over 40TB

– Smallest DB2 databases are 2TB with an average over 8TB

5

Why we chose Spectrum Accelerate (4/4)

It fits our workloads…

• Have very bursty and concentrated workloads

– Multitenant product – anyone at anytime can be “busy”

– Large read caches are very important – a full Spectrum Accelerate cluster can have

close to 2TB of memory

• We are very sensitive to read latency

– SSD cache is very important to keeping read response times under 2ms

– Typical query may require 10,000 or more 8Kb reads

– A full cluster has access to 12TB of SSD read cache – we know our hot working sets

will be on fast SSD while the cooler ‘historical’ data will remain on the less expensive

SATA drives

6

Get ready, get set, deploy!

• April: Prepared SoftLayer infrastructure

• July: 125 TB migrated to Spectrum Accelerate

• August: 270 TB “fully loaded”

• October: 500 TB in US, Canada, Europe

• October: 400 TB for another business unit

• November: Adding new cluster every 3 weeks

• 2016: Deploy 20 clusters in 7-8 more countries

Cookie cutter pattern! Deploy fast! Leverage SoftLayer IaaS!

April 2015

2016

Our Spectrum Accelerate solution in SoftLayer (1/3)

Hardware – standard SoftLayer 12 drive dual socket systems (Supermicro) with:

• 128GB RAM

• 2 x E5-26xx CPU (typically 2650s)

• 2x10GBps private network connections

• 1 800GB SSD – OS

• 1 800GB SSD Cache

• 10 x 4TB SATA drives

• ESX 5.5u2 – Leverage VMware Service Provider Program pricing from

SoftLayer for licensing

• Prefer LSI RAID controllers in JBOD but have had no issues with Adaptec

8

Our Spectrum Accelerate solution (2/3)

Size of clusters

– We like to build either 7 node or full 15 node clusters

– In SoftLayer we like to pre-provision a spare server

– We almost always provision all 12 drives per server

Why?

– Our smallest locations typically need about 100TB of space. Our SoftLayer

bare metal hardware choice provides us with about 120TB for a 7 node

cluster or 255TB for a full 15 node cluster

– Hardware will fail – we like to have a node ready

9

Our Spectrum Accelerate solution (3/3)

Features in use

• Obvious, but unique in a cloud environment – full host group & LUN masking

functionality.

• We have 1000s of LUNs in SoftLayer – XIV unified interface makes this

intuitive to storage and system administrators

• Replication – forming the basis for our geographical disaster recovery

leveraging the SoftLayer internal backbone (no cost for transit)

• SSD and Memory – we are seeing very good cache hit rates

• Performance tools – XIVTop versus limited or no visibility to cloud storage

options

10

Our Spectrum Accelerate solution: Workloads

• Oracle

– Primary databases and Data Guard copies

– Very high IOPs – on order of 70-100k IOPs per DB server

– 100+ LUNs per Oracle server in some cases

• VMware

– Providing primary storage for all VMDKs in our SoftLayer based private clouds

– We also do iSCSI LUNs mapped directly to Linux and Windows guest VMs

• DB2 on Linux

– ETL and DSS workloads – high read throughput

• Windows

– Clustered SQL server in SoftLayer

– Clustered Application Servers in SoftLayer

11

Best Practices: What we have learned about… (1/12)

• Deploying Tier 1 Oracle and VMware on Spectrum Accelerate

in public cloud

• Setting up SoftLayer

• Using VMware ESXi as the hypervisor

• Using Supermicro hardware

13

Best Practices: Client Hosts in SoftLayer (2/12)

One of our larger single Linux Oracle hosts in SoftLayer

• 130 assigned LUNs

• Over 250TB of assigned Spectrum Accelerate disk

• Using udev – managed by DBAs (well trained )

• Using HAK

• Sustained 1GB/s workloads

• 50-75k IOPs

• 40/60 Read to Write (~600MB/s writes)

• 4-6ms overall latency

• 70% IOPs @ 8kb

• multipathd: “ls /dev/sd* | wc –l = 801”

14

Best Practices: Client Hosts in SoftLayer (3/12)

• Leverage the XIV host attachment kit – it is excellent

– Team went from running a pure on-premises fiber channel storage

infrastructure to an iSCSI-based overnight

• Basic networking

– We use LACP on the host side

– Use iPerf to test, verify, and baseline your hosts and network configurations

– Tune the standard TCP tunables for Linux – net.core.*

• (See the old but still useful http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf)

– We have no issues in SoftLayer maxing out our 10Gbps links to Spectrum

Accelerate with the Supermicro AOC-STG ixgbe based NICs out of the box

15

Best Practices: Client Hosts in SoftLayer (4/12)

16

Typical day for one of our database servers in SoftLayer… 1GB/s

Best Practices: SoftLayer (5/12)

Networking

• Get your networking right – high volumes of traffic!

• Use large private portable networks for your iSCSI traffic

– We request and use /24 portable and primary subnets and keep all traffic

within a flat layer 3 segment: Avoid transiting network gateways!

– Add additional iSCSI subnets if needed

• Dedicated subnets for management and the interconnect IPs

– We use dedicated portable subnets for each

– No outside traffic allowed into the interconnect network

– Can apply network ACLs to our management network

17

Best Practices: SoftLayer + ESX – SA Hypervisor (6/12)

• Pay attention to interconnect and client traffic ports

– We are using an off-the-shelf SoftLayer configuration with two 10Gbps

private NICs per server

– We “break” bonding at the switch and at the ESX level

– Dedicate one interface to interconnect and one to client/management traffic

and set these to fail over for each other for the rare cases an access switch

or NIC fails

– Use jumbo frames – MTU of 9000 on client, ESX vSwitch, and Spectrum

nodes

• Leverage VLANs and Tagging

– Clients live on a default “app” VLAN and iSCSI VLAN is trunked in

18

Best Practices: SoftLayer + ESX – SA Hypervisor (7/12)

Hardware burn in…

• Allocate some burn in time for the SATA drives

• A full cluster will have 180 SATA drives of “various provenance”

• We find we will lose 2-3 drives in the first 1-2 weeks of a built out

• Rarely do we encounter a bad SSD, but we do check the smart data

• After 4 weeks our clusters achieve very good stability

– Bad disks have failed and any server issues (e.g. pink screens) have been

identified and rectified/replaced

– Run workloads against new cluster with your favorite disk IO tool

19

Best Practices: SoftLayer + ESX – Client Hypervisors (8/12)

We use Spectrum Accelerate to provide storage for our VMware ESX

clusters hosted on SoftLayer bare metal servers

• Create a single distributed virtual switch

• Trunk in our iSCSI VLAN to ESX server

• Create two iSCSI port groups

– A and B Side – each assigned to a single unique physical NIC

– Single VMkernel port on each

– Set single active uplink and disabled standby uplink on each

– Create software iSCSI adapter with the two iSCSI port groups

20

Best Practices: SoftLayer + ESX – Client Hypervisors (9/12)

21

Best Practices: SoftLayer + ESX – Client Hypervisors

(10/12)

We also have some “big” VMs

that actually connect directly to

our Spectrum Accelerate

clusters

• We cheat…

– Create a “storage port group”

– Assign single NIC to VM

– Expose 4 “target” paths to

VM’s iqn

– Let VMware load balance for

us

22

Best Practices: Using Supermicro hardware (11/12)

• Standardize on a firmware and driver for LSI

– Versions:

– Don’t trust SoftLayer to always deliver the same firmware – even within the

same order!

– Obviously watch the HCL for Spectrum with each version

However we have found it to be lenient on being slightly out of spec

• RAID Card

– OS disk is a proper “virtual disk”

– Cache SSD and SATA disks are in pass through/JBOD

– Typically have to “re-do” storage configuration on freshly ordered systems –

SoftLayer likes to deliver with all drives in a VD even if you didn’t ask for it

23

Best Practices: Using Supermicro hardware (12/12)

We really like to use the 12 drive 2RU systems

• Why?

– Our experience in SoftLayer has shown an order of magnitude fewer soft

and hard disk errors under load with the Supermicro BPN-SAS3-826EL1

back plane versus the larger BPN-SAS3-846EL1 back plan in the 24+ drive

systems

• What do we compromise on then?

– We can’t build maximum capacity clusters (1 OS + 1 SSD + 10x4TB)

– We use a single SSD for the ESX OS install and VMDK space for the

Spectrum Accelerate guest

24

Benefits in using Spectrum Accelerate

• Speed to deploy

– Cannot stress this one enough: leverage the IaaS of SoftLayer and the

power of Software Defined Storage

– 24-48 hours to spin up a 250 TB cluster – includes SoftLayer tasks

• Performance

– All performance benefits of XIV in our cloud environments

• Overall: Reliability, consistent performance, ease of use

– Cloud is naturally un-reliable – Spectrum Accelerate addresses that

25

Benefits in using Spectrum Accelerate

• TCO

– Can better match capacity to budget – no one shot at “capex”

– Deploy small cluster and add disks as needed – match expense to actual

load – less idle capacity

– Improve our Storage Admin efficiency – manage 2-3x the space per SAN

administrator

Important as cloud deployments tend to “balloon” footprints

– Have retired two physical data centers and one more is in our 2016 roadmap

– As faster servers, disks, and features come to SoftLayer and Spectrum

Accelerate we can immediately upgrade – no write-off/depreciation

26

What’s next

• Deploying our application in additional countries as SoftLayer opens

new data centers

• Continue to meet our growth – over 10TB per month of primary data

• Migrate all on premise legacy tape backups to disk to disk based

Spectrum Accelerate clusters in public cloud

• Evaluating moving to XIV in our current physical data centers

– Speed migration of products into public cloud

– Leverage replication for disaster recovery

– Leverage replication for dev/test at scale in public cloud

27

Summary

28

“Spectrum Accelerate is a Tier 1 array that we can

use on the cloud, spin up fast, and we know will be

reliable.

It gives us the right blend of speed to provision,

performance, operational simplicity and integration

with, again, just the whole ecosystem we are running

in SoftLayer.”

Paul Rafferty

Director, Infrastructure Architecture, Silverpop

Supermicro® Hyperconverged Appliance with IBM® Spectrum Accelerate™Speaker: Marc The’berge, Director, Rack Solutions, Supermicro

Email: marct@Supermicro.com; Twitter: @tbird960

The hardware below

• Silverpop use case involves:

– Spectrum Accelerate on SoftLayer® with underlying Supermicro hardware

– Future: XIV® integrated systems for fast data migration to SoftLayer

• Supermicro and IBM now offer a hyperconverged appliance with IBM

Spectrum Accelerate for your data center

– All the benefits of SDS for commodity servers and the ease and confidence of

a pre-tested, pre-integrated, validated appliance

– Proven enterprise-class storage capabilities for your data center and sites

30

Supermicro & IBM: Perfect synergy for a hyperconverged building block

31

• A leader in x86 infrastructure

• $2B/year publicly traded US

company

• WW sales and 24x7

service/support Manufacturing in

US, Europe and Asia

• First with dense servers and

storage

• Flexible and agile solutions

integration

• #1 in Software Defined Storage

platform by IDC

• $1B investment in SDS & Storage

• IBM Spectrum Accelerate is

based on proven XIV enterprise

storage

• #1 cloud computing provider in

the US by IDC

• #1 in US Patents for 22 straight

years

32

• Fast, low risk deployment

• Hardware, firmware, VMware® ESXi™ and IBM Spectrum

Accelerate stack configured and tested

• Complete appliance stress tested

• All hardware and cables labelled

• 6, 9 and 15 node standard configurations

• Compute, storage and networking cabled in the rack

• Customized configurations available

• With 3-15 nodes per instance

• More/different processors

• More RAM

• 800GB SSD; 1, 2, 3, 4 or 6TB NL SAS drives

• Customer choice rack, “rack ready,” or built into available

racks

Supermicro Hyperconverged Appliance powered by

IBM Spectrum Accelerate

33

Supermicro Hyperconverged Appliance powered by

IBM Spectrum Accelerate

Appliance SKU SRS-42USSG-SPAC-01 SRS-42USSG-SPAC-02 SRS-42USSG-SPAC-03

Node 6x 2U 12x 3.5" SSG 9x 2U 12x 3.5" SSG 15x 2U 12x 3.5" SSG

Processor/Node Intel® Xeon® E5-2620v3 Intel Xeon E5-2620v3 Intel Xeon E5-2620v3

Memory/Node 4x 16GB DDR4 ECC Reg 4x 16GB DDR4 ECC Reg 4x 16GB DDR4 ECC Reg

SSD/Node 1x 600GB Intel DC S3500 1x 600GB Intel DC S3500 1x 600GB Intel DC S3500

Total Data Drive 72 SAS3 108 SAS3 180 SAS3

Total Effective

Capacity114TB (4TB NL SAS) 186TB (4TB NL SAS) 330TB (4TB NL SAS)

Disk Controller LSI® 3008 SAS3 LSI 3008 SAS3 LSI 3008 SAS3

10G Interface 4x 10G SFP+ 4x 10G SFP+ 4x 10G SFP+

GbE Switch 1x SSE-G2252P 1x SSE-G2252P 1x SSE-G2252P

10GbE Switch 2x SSE-X3348SR 2x SSE-X3348SR 2x SSE-X3348SR

Notes:• Can be customized with 2nd processor, more powerful processor, more RAM, 1, 2 3, 4 or 6TB NL SAS

drives, 800GB SSD and 3-15 nodes per module• 3 nodes for test/development environments only, 4-15 nodes for production workloads

Who needs it and why: Use cases

34

Use case Who Why

Easy to manage

hyperconverged infrastructure

for VMware workloads

Small to large

businesses

• Powerful VMware integration lets VMware admins control

infrastructure, simplify deployment & management, avoiding

need for storage professionals

Development and test

environment/ ad-hoc projects

Medium to large

companies

• Fast deploy at lower risk as is tested and delivered ready to

deploy

• Need to save costs but want reliability and compute

• Can leverage in-house skills to build what else is needed

Powerful building block for

hyperconverged solutions

Medium to large

companies

• Fast deploy and at lower risk if using an appliance tested and

delivered ready to deploy

• Need to save costs but want reliability and compute

• Can leverage in-house skills to build what else is needed

Lower capacity enterprise-class

storage capabilities

Large companies

with many branch

offices

• Easy to centrally manage, replicate data between branch

office and headquarters

• Excellent price/performance and flexibility for backup and

remote sites

Hybrid cloud solutions Any • Increase use of public cloud in single managed on/off

premises environment

What we offer

• Two options:

– Rack option: Cabled and racked at the Supermicro factory

– Rack-ready option: Loaded with Spectrum Accelerate,

cabled and tested, with nodes, switches and cables labelled

and shipped individually. For customers who self-rack their

hardware

• Spectrum Accelerate enterprise-class capabilities:

– Scale-out, No tuning and no hotspots, management scaling

and mobility, integration with VMware, OpenStack®,

Microsoft® and IBM

• Support:

– 24x7, 4-hour response time from Supermicro hardware

support

– Software support from IBM and VMware

35

10GbE

Hypervisor

Spectrum

Accelerate

Spectrum

Accelerate

Spectrum

Accelerate

iSCSI

iSCSI

iSCSI

Hypervisor

Hypervisor

Supermicro Hyperconverged Appliance summary

• Perfect hardware/software synergy

• Appliance available with 3 standard configurations and custom

configurations

• Fast deploy for range of real needs

• 24/7 support

• Get one today!

36

Thank YouYour Feedback is Important!

Access the InterConnect 2016 Conference Attendee

Portal to complete your session surveys from your

smartphone,

laptop or conference kiosk.