architecting, developing, and optimizing a virtual bi 4.0 ... · [architecting, developing, and...

[ Architecting, Developing, and Optimizing a Virtual BI 4.0 Data Center

Ashish C. MorzariaDirector – Solution ManagementCloud, Virtualization, and Appliance StrategiesSAP Business Analytics

Real Experience. Real Advantage.

[ Disclaimers and Legal Stuff

Session is for education only:Overview of technologies and factors that will affect you in bringing enterprise applications such as BI (and even EIM, ERP, or CRM, etc) to virtualized environments

Your mileage may vary:Not a substitute for proper education, careful analysis of your own environment and serious discussion with people who can help and support you (including SAP!)

This presentation is for education only and should not be relied on in making a purchase or configuration decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.


[ Speaker Introduction

Name Ashish C. Morzaria

Role Director - Solution Management, Business Analytics, SAP

Topics Hybrid Cloud, Virtualization, Appliance Deployment ModelsBI Suite Solution Management

Background & MotivationSeven years at SAP/Business Objects in Product and Solution Management - previously led Business Objects’ Search Portfolio, Mobile BI, OEM, and SAP StreamWork Enterprise product lines.

Developed and released SAP StreamWork - SAP’s first (and only) commercially supported virtual appliance-based product in 2010.

Currently working with a number of SAP teams to deploy more SAP products in the Cloud, and in virtualized and appliance form factors


[

This session will assume you know what virtualization is and what it is good for – the fact you are here means you want to know how things work underneath and don’t just want your BI to run virtualized - you want it to run virtualized well!

4

Special Notes Of This Webinar

This content is dated and abbreviated:Content from mid-2011 – Quite a bit of learnings since then!

*Most* of this content is still relevant – some previous content cut

Nature of webinars makes it difficult to gauge audience, we’ll go a bit slowerMain point: you need to understand virtualization to virtualize the BI suite

Upcoming sessions:SAP Cloud and Virtualization Week: April 18th:

“SAP BI In the Cloud – Where It Makes Sense And Where It Doesn’t”

SAP Sapphire/ASUG Conference: May 14th:“Virtual Reality: Architecting Virtual BI 4 For Production Environments”


[ Agenda

What’s “in” for today:Virtualization for enterprise-grade production BIArchitecting a virtualization stack:

Understand each layer and their effects

Diving into the technology:CPU, I/O, and RAM issues (i.e. scheduling, context switching)

Host and guest considerationsChoosing, configuring, and optimizing

Running SAP in virtual environmentsIssues to consider and support statementsRecommendations for SAP software (focus on BI)

What’s “out”:The optimal “answer” – that’s for you to decide!Which specific products to use in your stackOS and app specific tuning

A separate session required for real value


[

VIRTUALIZATION FOR PRODUCTION BI ENVIRONMENTS


[

7

Issues With Virtualizing BI (1/2)

I/O is the constrained resource:Nature of today’s BI is increasingly moving from CPU to I/O and RAM intensive workloadsHardware (CPU/RAM) are physical “things” that can be provisioned, metered, and constrainedI/O is a process which is difficult to monitor or restrict efficiently

Virtualizing I/O intensive applications is actually quite difficultBI is not a “simple” application like Web serving, DNS, etcDissecting performance and reliability issues of production apps in a multi-tenant environment is difficult, dangerous, and fraught with problems

Virtualization tools for I/O are still in their infancy:Focus until now is on monitoring/optimizing CPU and RAMTools are needed to “monitor the monitor” (for the hypervisor)


[

8

Issues With Virtualizing BI (2/2)

Product concerns:Are the BI applications designed/certified/tested to be virtualized?Do applications perform very differently (i.e. poorly) when virtualized?

Concerns with support in virtual environments:How will functional issues be addressed in a virtual environment?Will performance issues be addressed by the software vendor?Will you be required to reproduce issues in a physical environment?

Licensing issues:CPU-based pricing sometimes doesn’t make sense in a virtual worldOver-committing CPUs decreases HW cost but increases SW costScaling out makes more sense but can significantly increase costs

Virtualization enables you think about deployments differentlyShould you scale up or scale out?


[

9

A Recipe for “Optimal”?

Virtualization is as much “art” as science:Incredibly complicated - simulating general purpose hardware is tricky because we don’t know how the hardware will be usedToo many parameters changing variables too quickly for humansThe best “theory” is no match for a real server hitting “the wall”

Nobody understands your environment but you:Your usage scenarios will dictate how to design the systemExternal (to virtualization host) environment is constantly changingPost-install monitoring and adjustment is absolutely necessary!

Technology changes every 6-12 months:New kernels, new CPUs, and new hardware come at random timesPerformance delta to physical hardware is shrinking with new features added to existing virtualization stacks


[

ARCHITECTING A VIRTUALIZATION STACK


[ Administrator’s View Of Virtualization


[

12

Stack – Hardware - CPUs

Getting the right hardware is critical to a good deployment:In general, the most recent CPU generation will have better virtualization performance – even at the same clock speedMore recent “steppings” of a CPU will have better overall performance –even within a specific chip family

Multi-socket systems are better than single ones:More cores enables you to better match vCPUs with pCPUsSystems typically handle more RAM (less need to over-commit)Systems typically have more I/O (more lanes, more bandwidth)Note: Multi-socket systems may use cores with lower clock freq

Goal is to manage performance vs. power consumption (and heat)In most applications, a better vCPU:pCPU ratio is better than having faster cores


[

13

Stack – Hardware - Physical

For new projects, don’t settle on sub-optimal hardware:>1 year old is sub-optimal hardware but capable>2 years should not be used for new, long-running projects/apps

Hardware support by hypervisor is important:Ensure that your choice of hypervisor supports the hardware – down to the device and driver levelsSome HVs are stripped down and may support motherboard chipsets “generically” rather than with optimized drivers

Your choice of hardware vendor is important:What support will you get (if any) for virtualized workloads?Hardware vendors have a lot of experience in provisioning the right hardware for the right workloads (but don’t drink the Kool-Aid!)


[

14

Stack – Hardware – I/O

Storage - local storage, iSCSI, FC, or other?Evaluate I/O loads and ensure bandwidth is always available

Ensure that your SAN or storage servers are not oversubscribed!

Local storage is less desirable except for specific situations (i.e. swap)Some HV features require SAN storage (i.e. VMotion)

Networking – what type, how many NICs, and which features?Improve performance through NIC teaming and TCP offloadingIncrease reliability through NIC teaming through multiple switchesAllocate one physical NIC per VM – greatly reduce context switchingDedicate one NIC for hypervisor managementCovered later in this session:

Don’t forget the use of para-virtualized NICs in the VMs whenever possible


[

15

Stack – Hypervisor – Configuration

Default hypervisor setup is to “get up and running”: No optimization based on workload, OSes, I/O, or host H/WNo hardware-specific drivers that can improve overall performanceMost defaults will weight all VMs equally in all respectsNo special features will be enabled by default

Do not rely solely on the hypervisor for monitoring:Easy to monitor CPU and RAM – but how easy is it to manage I/O?

You cannot use the hypervisor to report on the hypervisor

Explore the third party offerings for monitoring, alerting & backupSome are free, some are low-cost: check out vKernel for VMWare systemsDon’t believe systems are optimized once you hit the “fix it button”

Without proper education, technical tools will make no sense


[

16

Stack – Guest Operating System

Modern (2-3 yr) OSes are better guests:Memory allocation and CPU scheduling have been re-architected:

HVs rely on memory compression, lazy allocation, over-committingGuest schedulers can be messed up when its CPU cycles get restricted or rescheduled by the hypervisor – newer OSes treat memory differently

Windows or Linux? Very little difference in performance between:SuSE Linux Enterprise Server 11Red Hat 5 (including CentOS, Oracle Linux, etc)Windows 2008 R2

Choice of hypervisor may be impacted by your OS:VMWare has by far the widest support, followed by CitrixSLES is a “best guest” on Microsoft

RHAT runs, but not supported

Are your OSes properly and fully supported by your HV?Check out the “guest tools” to ensure proper support

Friends don’t let friends run old OSes in virtual

environments except when required


[

DIVING INTO TECHNOLOGY


[

18

Hypervisor Are “Masters Of The Universe”

Hypervisors manage all hardware interactions:Gathers and schedules instructions from each VM to feed each CPUManages interaction with disk and network through shared connections

Time slicing is still a fundamental element of virtualization today

Hypervisors software elements too:Management, monitoring, and configuration of each environmentEven the hardware clock is managed (emulated) by the hypervisor!Guest tools enable the guest to “co-operate” with the HV

“The landlord has the key to the house, but doesn’t come in very often”


[

19

Hypervisor Context Switching Explained

Context switching:Switching of the CPU from one process or thread to another. The state (context) is stored so other processes can be run before restoring the current state to continue executionDefinition can extend to almost any shared process or physical resource (i.e. CPU, disk, network)

Switch from Process A’s context to hypervisor’s

Switch from hypervisor’s context to Process B

Hypervisor runs between Process A and Process B

Hypervisor


[

20

Overloading CPUs Increases Context Switching

Hypervisors prefer to leave resources committed:Lazy = OptimizedWhile the HV schedule may issue a HALT to the CPU so it can process another CPU, it may not require a full context switch (i.e. from A->B)You can help the hypervisor by setting processor affinity

When vCPUs > pCPUs, a context switch is very likely:By definition, the HV requires a context switch to handle a different VMBut processor affinity now also works against you:

You guarantee sharing pCPUs, but you restrict the HV from making the best choice of which pCPU to use

What about the Hypervisor itself? Does it need a CPU too?Dedication doesn’t apply because the hypervisor runs across all CPUs


[

21

How Many vCPUs Are Enough?

Does a multi-vCPU VM do better with more or less cores?

Less!Two schedulers at work – guest OS and HVGuest VM scheduler may try to parallelize instructions to use all CPUs

HV thinks all CPUs are required to execute (negating its own scheduling)Result: HV starves busy CPUs so all are idle at the same time

Does that mean 2x 2 CPU VMs is better than 1x 4 CPU VM?Yes – but you should be looking at this for other (better) reasons too!Scaling out is almost always a better solution than scaling up CPUs (>4)Moving to better hardware is the best way to improve performance of a VM without adding more CPUs


[

22

Context Switching Affects I/O Too

Every access to the network requires context preservation:Network I/O can be a killer – two I/O bound VMs sharing one NIC saturates bandwidth and forces the HV to context switch oftenMore HV mgmt = more CPU = more overhead = lower VM performance

Disk I/O requires special thought:I/O capacity can be easily saturated if multiple VMs hit it at onceHVs are good at scheduling/reordering I/O, but it requires work!More HV work = more CPU = less performance = lower I/O

How do I reduce I/O context switching?Dedicate one NIC to HV for “management network”Dedicate one NIC per VM when possiblePlan your I/O paths carefully – is there one SAN link from the host?


[

23

Hypervisors And RAM

Hypervisors are excellent memory managers:Most provide RAM compression to provide more memory to other VMsMemory over-commit is technically impressive and works really well

Financially, maybe not with new licensing models (i.e. VMWare 5’s model)

“Guest Tools” are critical:Main component includes vmemctl – para-virtualized RAM driverWithout the tools, HV has very little visibility and must rely on the guest’s context to manage memory and can only guess how to optimize

Memory reservations are important in mission critical servers:RAM must always be available - the alternative to it is disk I/O!Some applications (including SAP NetWeaver) sparsely reserve RAM

Hypervisors may see this as the VM using less RAM than the application thinks it does – over-committing (and swapping) is almost guaranteed


[

24

Swapping RAM – The Silent Performance Killer

Swapping - the ultimate context switch:If context switching on and off a stack is a thousand paper cuts, swapping RAM to disk is cyanideHV gets hit two different ways:

Context switching and wait times for RAM that isn’t thereManaging I/O for the actual swap(And “bonus” – if the VM is already swapping internally, this gets really bad)

You can reduce swap issues by choosing locations carefully:Swapping across a SAN is the worst, followed by swapping locally

Swap is a delicate balance between RAM and disk allocation:Allocating more RAM costs more, but greatly improves performanceFor mission critical applications, this shouldn’t even be a question

Make the best use of your application and assets – don’t starve the VM


[ Memory Mapping In Virtual Environments

Example of memory mapping:Over-committing is very useful, but you must decide if it is truly worth itSwapping the host’s machine memory blows the SLA for all VMs

There is nothing you can do when the SLA-enforcer can’t meet its own SLA


[

VIRTUALIZATION HOST CONSIDERATIONS


[

27

Optimizing Host CPU And RAM

Avoid over-commits:Resources are never created, they are shared!Use as few virtual CPUs per VM as possible

VMs scale well to 4 vCPUs/32GB, or 8 vCPUs in CPU-bound applicationsYour mileage may vary depending on the application used

Don’t make promises your hypervisor cannot keep!The penalty for production servers are rarely worth the saving from over-committing CPU or RAM

But, But, But…My host utilization is really, really low!

Benchmark acceptable performance before adding new loads to this hostHost may be over-provisioned on CPU/RAM but under-provisioned on I/O!

The Host reports each VM is using a fraction of its RAM!Consider cost of additional RAM vs. performance bottlenecks as system usage growsConsider VMWare’s new RAM-based licensing model for VMWare 5


[

28

Guest VM Configuration

Guest tools are absolutely important!Alarming number of environments without guest tools:

No VM awareness/co-operation with hypervisor or vice versaNo monitoring or management capabilities (even time sync)Hypervisor is forced to context switch far more than necessary

Virtualization features may not work well or at all without tools:No optimized drivers (emulating generic hardware w/many context switches)No para-virtualized drivers

Use para-virtualized device drivers when possible:VMs may require boot drives to use non-para-virtualized driversMultiple options for para-virtualized NICs – so consider the use case

Enable any OS/guest specific optimizations:Don’t forget non-virtualization optimizations – they apply here too!


[

29

Does Everything Have To Be Virtual?

Virtualize only where it makes sense!I/O intensive components of mission critical systems should stay on physical servers whenever possibleSome I/O-focused servers do not like VMotion-type movements

Special care needs to be taken when clustered file systems are involved

CPU-based licensing may affect your virtualization choice:If you are paying by CPU, can you afford to lose 10-20% performance?

RAM-based licensing may introduce new costs:VMWare 5’s RAM based licensing can increase your total cost if you don’t use RAM over-commit featuresNext generation “In-Memory” products absolutely require dedicated physical RAM (i.e. SAP’s in-memory analytics platforms)


[

RUNNING SAP IN VIRTUAL ENVIRONMENTS


[

31

Business Objects BI Considerations

Business Intelligence is even more I/O intensive than ERP:Stresses to I/O even more critical (and harder to measure than CPU/RAM)Aggregating millions of rows is very different than streaming transactionsSpiky load makes estimation even harder – constant load vs. peak timesUnderscores importance of understanding the workload *before* you start

Virtualization benefits are (potentially) even higher than ERP:Multi-node clusters work better than massive, monolithic systemsHigh availability features from HV provides an extra layer of HA at the H/W levelCloning, snapshots, and environment replication is easier and makes more sense in a relatively stateless environment vs. a time critical time based system


[

32

Enterprise Class BI In Virtual Environments

SAP software is typically in the “mission critical” category:Virtualization introduces new factors to an already complex systemCritical to follow SAP’s guidance on virtualization environments and configuration for the software to deliver value with performance

Deploying for demo / test / trial / training environmentsSLAs do not apply – typically “unsupported” installations/configurationSupport is on a best efforts basis as technical requirements on customer for SLAs not enforced

Production use:SAP fully supports - customer must follow requirements/guidanceCustomer has freedom to use VMWare, Xen, or Hyper-VKVM support has limited support by SAP, but NOT for BOBJ apps:

SAP Note 1522993: SAP on SUSE KVMSAP Note 1400911: SAP on Red Hat KVM


[

33

SAP’s Virtualization Support Statement

Standardized support statement for all (most) SAP products:SAP Note: 1492000

https://service.sap.com/sap/support/notes/1492000

Includes most “SAP Classic”, Business Objects, and Sybase productsSome exclusions apple (i.e. SAP HANA, BWA, GRC)

Links to other notes for x86 and different platforms (i.e. AIX, Solaris)

Support for functional issues:Identical to physical instances b/c transparent nature of supported HVsHV vendor supports for “non-transparent’ issues raised

Support for performance issues:Requires the setup of a central monitoring instance in SAP Solution Manager or in a suitable SAP NetWeaver ABAP systemPerformance issues related to hardware, hypervisor, and guest VMs are the customer’s responsibility


[

34

Business Objects BI Considerations

Expect overhead of ~20% for a typical deployment:Could be lower - but remember I/O overhead has a larger impact on BIFactor overhead into everything – including installations, operation, patches, updates, etcScheduled processes (i.e. AV scans) running on multiple guests on the same host will hammer the host – stagger the scans whenever possible

CPU, I/O, and disk context switches at the same time hurt:BI uses all three – overhead gets bigger as resources get hit togetherIncreasing “ready times” if multiple resources are needed at once

“Scale out” instead of “scale up”Adding more resources can actually make things worse

Create multi-image BI deployments vs. a monolithic systemAdding more vCPUs will usually not improve performance (>4)Best performance with 2 vCPUs (not more than 4 per VM)


[

35

SAP Virtualization Recommendations

Maximize your resources:Provision as much RAM as required to avoid guest or host swappingUse resource reservation whenever possible (CPU, RAM, I/O) Set min reservations per VM to ensure one doesn’t degrade anotherCreate resource pools to better manage resource usage/constraintsUse hypervisor “priority shares” to weight VMs properly:

Determine which VMs should get priority when the server “hits the wall”Not all VMs should have the same priority – if they do, you might not be spreading the load across hosts well enough


[

36


Maximize disk I/O:Use para-virtual SCSI drivers whenever possibleUse resource pools wherever it makes senseUse local host storage for host swap partitionsConsider dedicated I/O links for VMs that can saturate a single linkTier your data stores by performance – not all iSCSI sources provide the same performance, cost, or features


[

37


Maximize your network I/O:Use para-virtual network drivers whenever possibleMap physical NICs to virtual NICs 1:1 for best performance

Many to one vNIC to pNIC mappings can quickly saturate the host

Dedicate a separate NIC for the hypervisor management networkUse multiple virtual switches if you have a lot of inter-VM communication


[

38

Virtualizing Do’s And Don’ts

Do’s:Understand what it takes to be supported by SAP in productionGet training, education, and support before starting the projectUse the latest CPUs and don’t scrimp on the RAM or SAN storageUse affinity and reservations to improve performance of mission critical enterprise applications by reducing management overheadUse guest VM tools – even for dev/test/training systemsOptimize the guest – disable services, reconfigure schedulers, etc

Don’ts:Thin provision resources (i.e. virtual disks should be pre-allocated)Compromise SLAs on mission critical systems for a few bucksJustify moving SAP systems to VMs for server consolidation *only*


[

CLOSING THOUGHTS


[

40

SAP’s Roadmap And Virtualization

“In-Memory” technologies are game changers:Ability to analyze hundreds of terabytes of streaming data in real-time with sub-second response timesCurrent virtualization technologies are based on traditional software deployments and coding techniquesWhen computation is done real-time (constantly) and calculations are triggered with every memory access, resource usage skyrockets

SAP is leading the way in new virtualization techniques:Deep collaboration with VMWare, Citrix, Microsoft, SuSE, Red HatDedicated Linux Lab co-located with all hardware and software partnersDevelopment collaboration with leading cloud vendors (i.e. Amazon)

“In-Memory” techniques require access to large memory spaces (up to 2 TB RAM) – The value proposition of virtualization is very different because the “rules of the game” have changed


[

41

Key Learning Points

You can run SAP applications virtualized in production TODAY:

SAP Note 1492000 provides standardized support over many hypervisorsSAP has specific guidelines to help optimize performance for productive use

Understanding the mechanics of virtualization is important:If you want enterprise-class performance and stability, you need to invest in time, training, tools, and testing a new platformYou will make better choices if you understand their implications

Overhead can be as much as 60% or as little as 10% - it’s really up to you!

Context switching and I/O can dramatically affect performance:Context switching is a fact of life, but can be minimized if you pay careful attention to how the system works and how you want it to behaveI/O is the least measurable (and therefore least understood) of all virtualization factors – but it can silently poison your host without warning


[ ASUG Annual Conference – May 13-16, 2012

42


[

43

[]Thank you for participating.

Architecting, Developing, and Optimizing a Virtual BI 4.0 Data CenterMarch 29, 2012

Please remember to complete and return your evaluation form following this session.

For ongoing education on this area of focus, visit the Year-Round Community page at www.asug.com/yrc

Ashish C. [email protected]

architecting, developing, and optimizing a virtual bi 4.0 ... · [architecting, developing, and...

Documents