CAS91545
Huge client SAP landscape transformation
to SAP HANA on IBM Power Systems with
SLES in private cloud
Carsten Dieterle
IBM Germany
2
Carsten Dieterle
Leading SOLAR(SAP Infrastructure SOLution ARchitect)
Member of the IBM Technical Expert Council (TEC) DACH
IBM Deutschland GmbH IBM-Allee 171139 EhningenGermany
Phone/Mobile +49 7034 274 [email protected]
Agenda
• SAP HANA on Power Introduction
• IBM Cloud Orchestrator in a cloud environment
• SAP HANA history at client
• Test and Evaluation Phase (TAE)
• SAP HANA on Power Project (Power E880, ESS, ICO)
• next steps (HA / DR setup) and SAP System Replication
3
SAP HANA on IBM Power – From Start till To-Date
4
History - Development activities
August 2013
SAP initiates Product Development Program for HANA on IBM
Power technology
December 2013
HANA code optimized compiled/linked (HANA SPS08,
SLES11SP2)
February 2014
All HANA Servers running, IBM code checked in, full function
testing commencing, Power 8 optimization agreed
June 2014
SaphireNow/Bernd Leukert announces “Test and Evaluation
Program” for selected customers in 3Q 2014
October 2014
SAP TechEd && d-code/Bjoern Goerke announces HANA on
POWER SAP Ramp-Up Program” beginning 1Q2015
November, 2014
BW 7.31+ TEA code ships to customers
March, 2015:
SAP Ramp-up opened for customer applications
July 2015:
SAP Ramp-up successfully closed
August, 2015:
GA SAP HANA on IBM POWER
December 2015:
Support for SAP Suite on HANA and
HANA Platform Edition SPS 11
May 2016:
Support for S/4HANA
SAP HANA on Power Tailored Data Center Integration
5
• Solution validation done by SAP and
partner
• Preconfigured hardware set-up
• Preinstalled software
• Installation needs to be done by customer /
responsibility of customer
• Customer aligns with the hardware partner on
individual support mode
Fast Implementation
Support fully provided by SAP
More Flexibility
Save IT budget and existing investment
Appliance approach Tailored Data Center approach
SAP HANA on IBM Power (Status: 08/2016)
SAP HANA on IBM Power supports: • SAP ERP 6.0 EHP7+8
• SAP SRM 7.0 EHP3+4
• SAP CRM 7.0 EHP3+4
• SAP SCM 7.0 EHP4
• SAP BW 7.31, 7.40, 7.50, BI JAVA 7.40 for SAP HANA
• S/4HANA (native install starting with 1511 SR1 on prem),
e.g. S/4HANA 1511 FPS 02, S/4HANA Finance 1605 and S/4 SFinancials 1503
• and a lot of other products on HANA on Power (e.g. BPC 10.1, SAP BO BI 4.1 SP03, CAR 1.0 SPS03, MDG 7.0, Banking 9.0/BA
(Bank Analyzer), SolMan 7.2, SLT 2.0, SDI 1.0 SP02, Hybris Commerce / Marketing, etc.)
Core to Memory ratio (per productive HANA LPAR): – higher ratio potentially on individual request@SAP:
please see SAP Note 1903576 - SAP-HANA-Database: additional main memory in exceptional cases
• BW and other, not exactly specified workload• 50 GB per core on E880 (max. 96 cores; at most 4800 GB RAM)
• 50 GB per core on E870 (max. 80 cores; at most 4000 GB RAM)
• 32 GB per core on E850 (max. 32/48* Cores) and S-models
• SAP Business Suite (SoH: Suite on HANA), S/4HANA, SFINANCIALS, Banking (including SAP Bank Analyzer), SAP
Solution Manager, SAP Master Data Governance workload• 96 GB per core at most 9 TB RAM
• Minimum core count and min. memory per HANA DB LPAR:Production: 4 cores, 128 GB Memory - Non-production: 2 cores, 64 GB Memory
BW Scale-Up & Scale-Out
(up to 16x 4800 GB, xfs based
or SpectrumScale/GPFS based ESS appliance)
SoH Scale-Up
http://service.sap.com/sap/support/notes/2218464
http://service.sap.com/sap/support/notes/2188482
*) with new models E850 with feature codes EPW5 and EPW6
6
SAP HANA on IBM Power (Status: 08/2016)
Multiple HANA LPARs supported for production on a single server:
up to 4 dedicated or 3 dedicated and 1 shared pool
Support for Clustering Support - HANA BW Scale-out(up to 16 nodes / up to 4800 GB each, xfs or Spectrum Scale/GPFS based ESS appliances)
HANA Platform Edition (SPS11, SPS12 etc.) availability for POWER (at the same time as with x86)
Non-Production relaxations of the requirements for SAP HANA on IBM Power Servers
• POWER8 and POWER7+ servers allowed
• No restrictions with regards to the maximum memory per HANA LPAR core
• The minimum core count per HANA LPAR is 2, min. Memory is 64 GB.
• HANA LPARS can be run in a shared processor pool running with a performance penalty
• Standard Network Components are admitted.
• SAP HANA LPARS for test, development and quality assurance systems may run in parallel to
other LPARs (for SAP HANA or other purposes) on a single physical Power Server as long as
none of the participating LPARs runs a production SAP HANA system.
• Besides xfs also NFS, GPFS and ext3 filesystems are supported.
7
One (Hybrid) Cloud Orchestration tool for different platforms
“POWER Cloud”
“x86 Cloud”
“z/VM Cloud”
“Storage Cloud”IBM Cloud Manager
with OpenStack
“Public Cloud”
© IBM Corporation 9
IBM Cloud Orchestrator – Key Capabilities
Composable and integrated application development platform
Pattern Management
Enabling business transormation
Self Service Catalog &Orchestration Workflows
Service Orchestration
Workload Orchestration
Infrastructureas a Service
From a capability point-of-view, IBM Cloud Orchestrator can be represented as a three-tier solution
Software Defined, Enterprise class, optimized infrastructure
Infrastructure ManagementNetworkingCompute Storage
• Deploy workloads to multiple virtualization infrastructures and public clouds.
• Configure Compute, Network, Storage resources.
• Automate deployment of whole application, via blueprint
• Composable multi-node patterns
• Automate everything for production-ready
• Workflow engine to automatically configure policies (backup, patching, monitoring)
• Integrate with service management tools
Self service, multi-tenant platform.
Public CloudsPrivate Clouds
7
© IBM Corporation 10
Cloud Orchestrator Use Case: Self Service IT Delivery
Pain Point:
• It takes weeks-months to put a ready-to-run business application system into production with
many different tools / teams
• LOB creating their shadow IT, by-passing Ops. With no goverance and control
Customer value:
• Standardization of application infrastructure via multi-tenant catalog–based delivery
• Deliver IT services to LOB in minutes. Same speed as their shadow IT, but in full control.
• Enable choice of best on-premises as well as off-premises clouds: design the service once, deploy
everywhere
Cloud Orchestrator Use Case: Self Service IT Delivery
8
SAP and SAP HANA history at client
11
• Client is using IBM Power servers with Oracle DB and IBM and non IBM storage for a huge number of SAP installations (ca. 500 SAPSID’s)
• Since early 2013 client installed SAP HANA appliances from IBM and HP
• 2014: Client installed a SAP BW 4 Node scaleout cluster from IBM
• Client interest about SAP HANA on Power was extremly high, participation in TAE for HoP, installation of HoP in november 2014
• April 2015: end of TAE, Feedback to SAP, RampUp begins and client migrated first SAP SID to HoP
• RfP for SAP HANA infrastructure Q3 / Q4 2016
• HoP installation in a private cloud environment starts in Q1/2016
HoP setup from TAE Program @ client: SAP BW on HANA, 1 TB HANA-DB-TDI-Server
12
TAE Installation System 8286-42A
S824 Power8
1 x 8286-42A
1024 GB RAM
24cores @ 3.52 GHz
4 cores / 8 GB reserved for Dual Vios Setup
15 cores / 768 GB for SAP HANA
SLES 11 SP3 with patch
XIV Gen 3
2x
2-p
ort
8Gb FC
attachment
S824 / PowerVM
Linux – 1024 GB / 24cores
SAP HANA DB
Recommended: TDI certified
SAN Storage
1Gb-ET
4-ports
2* 10 Gb-ET
4- ports
First experiences from Test & Evaluation Program
• Migration from HANA x86 4 Node Cluster á 512 GB (3 active, 1 standby)
to HANA on Power S824 with 1 TB.
• XIV Gen 3 (without SSD's) as a HANA TDI certified storage
• Migration with standard SAP heterogeneous system copy (migration),
SAPinst / R3Load
• Installation and SAP migration showed no issues
• For these “first of a kind” installation you need a Linux specialist
• Supporting SAP Solution Architect (Solar) for this case was Carsten
Dieterle
13
First experiences from Test & Evaluation Program
14
Filesystem layout
XFS with multipathing
• Minimum of 4 active paths for performance
• Beware of zoning and storage subsystem specifics
LVM striping required for performance
Minimum of 4 volumes for each data and log
Number of volumes should be a multiple of the number of
active paths
15
Solution: The SAP HANA on Power private Cloud @ client
14
Enterprise servers E880
“Compute Cloud”
RHEL
Linux
IBM Cloud Manager
with OpenStack
IBM ESS
“Storage
Cloud”
ESS Model GL6
Userclient Intranet
CIoud Adminincl. ICO Request
„SAP LoB“ User
SAP HANA System Requests
Workflows
Application Overview Diagram (high level, production HANA) with ICO
15
E880 #1
VIO1 VIO2
E880 #2
VIO3 VIO4
PowerVC
HMC
IBM Cloud Orchestrator(ICO 2.5) Virtual Maschine
workflows
P01
GPFS
ESS
self service
CMDB
interfaces(provided by
client)
on VMware platform(provided by client)
hardware offered by IBM & partners
P03
P01*
P04
P02* P03
T01 T02 ... T79
InfiniBand
Setup for technical proof of this first of a kind setup / architecture
(as is May / June 2016)
16
Productive HANA on Power LPAR Setup
17
SEASRIOV Ethernet
8GbFC
LPAR SLES 11 SP4 3TB
FDR InfiniBand
IBM ESS GL6FDR Infiniband
VIO Server
Boot disksvNIC
V3000 StorageV3000 Storage
IBM ESS GL6
Legend
Virtual function with LACP
SEA
InfiniBand Switch
Ethernet Switch
SAN FC Switch
Link Aggregation LACP
First Virtual Function SRIOV
InfiniBand Adapter
NPIV SAN Interface
SAN FC Adapter
10GbEth
NPIV vNIC
Non productive HANA on Power LPAR Setup
18
SEA
8GbFC
Non productive LPAR SLES 11 SP4 512 GBIBM ESS GL6
VIO Server
vNIC
V3000 StorageV3000 Storage
IBM ESS GL6
Legend
SEA
Ethernet Switch
SAN FC Switch
Link Aggregation LACP
Second Virtual Function SRIOV 10% of 10 Gb/s
NPIV Boot disks
SRIOV EthernetVirtual function with LACP
10GbEth
vNIC
V3700 #2Citrix
FW
ESXi #1
ESXi #2
RDP/VMC OS?Host
VM ICM 4.3
ICO 2.5Supp xxxVM RZ1
Power VCVM
880 #1
880 #2
VIO
IB
SIDSID
SID
LPAR (HANA/Prod)
Service IP
Public IP
LPAR /HANA Non-Prod)
vSCSI
Public IP
SP
SRI0V
GPFS/Ethn
GPFS/IB
HANA Access
Frontend LAN
IB
ESS#1
10
HMC
HMC
HMC
VIO
FCFC
Admin/1Gb
IPPool
Web Browser+ Java
ESS#2
FW = Firewall
Userclient Intranet
IT AdminIncl.. ICO Request
„SAP LoB“
RHJ1
Info
blo
x
Mgmt IP
Public IP
HMC
ESS#3
#2
#2IB Switch#1
FC Switch#1
FibreChannel
Infiniband
#2Switch#1
V3700 #1
SSP
FW
AS
FW
19
Solution Design I for IBM Power Servers
SAP Business Warehouse (SAP BW): Scale-out across multiple E880 servers
• 6 LPARs on 6 E880 running SAP BW powered by SAP HANA
- 3 TB memory each
- a core to memory ratio of 64 GB per core
- Operating System (initial) is SLES 11 SP4
- Linux boot disks are SAN attached (via VIOS and vFC)
16 TByte SAP BW: 5 worker nodes, 1 standby node, distributed across 6 E880 systems
• Each E880 is equipped with
- SR-IOV capable network adapters,
- Infiniband adapters
- FibreChannel adapters
- two VIOS LPARs
• Filesystem for SAP HANA is GPFS
- Storage Servers are two ESS GL6
- Connectivity between SAP HANA (Spectrum Scale Client) and Storage Server (Spectrum Scale Server) is preferable
achieved via Infiniband
IBM Power E880 (154 cores / > 12 TByte)
Fill-up E880 with SAP BW Prod-Systems (Alternative ONE)
(Core-to-memory-ratio: 1 core per 64 GByte Memory)
21
64 GByte/ core (max. 48 cores, max 4 TB)
100% 75%
3 TB
SAP HANA(48 cores / 3 TB)
VIOS 1
5 cores
VIOS 2
5 cores
E880 Building Block (max. 4)
3 TB
SAP HANA(48 cores / 3 TB)
3 TB
SAP HANA(48 cores / 3 TB)
IBM Power E880
58 cores / >3 TByte
IBM Power E880
58 cores / >3 TByte
IBM Power E880
58 cores / >3 TByte
IBM Power E880
58 cores / >3 TByte
IBM Power E880
58 cores / >3 TByte
IBM Power E880
58 cores / >3 TByte
HANA 1Host AutoFailover
5 worker
VIOS 1
5 cores
VIOS 2
5 cores
VIOS 1
5 cores
VIOS 2
5 coresVIOS 1
5 cores
VIOS 2
5 cores
VIOS 1
5 cores
VIOS 2
5 cores
VIOS 1
5 cores
VIOS 2
5 coresVIOS 1
5 cores
VIOS 2
5 cores
Initial Configuration – 6 E880, One 16 TB SAP BWAt least 58 cores / 3 TByte of Memory are required per E880
22
SAP HANA Cloud 6 IBM Power E880 can host three 16 TB SAP BW Systems
23
IBM Power E880
154 cores / >12 TByte
IBM Power E880
154 cores / >12 TByte
IBM Power E880
154 cores / >12 TByte
IBM Power E880
154 cores / >12 TByte
IBM Power E880
154 cores / >12 TByte
IBM Power E880
154 cores / >12 TByte
HANA 1
HANA 3
HANA 2
Host AutoFailover
5 worker
Host AutoFailover
5 worker
Host AutoFailover
5 worker
Simplistic view:
3 HANA Systems with same topology
provisioned to IBM Power Server farm
Flexible HANA System sizes
(number of scale-out nodes,
node sizes, different core-to-memory
ratios) are possible – the servers then
are filled with various building blocks!
VIOS 1
5 cores
VIOS 2
5 cores
VIOS 1
5 cores
VIOS 2
5 coresVIOS 1
5 cores
VIOS 2
5 cores
VIOS 1
5 cores
VIOS 2
5 cores
VIOS 1
5 cores
VIOS 2
5 coresVIOS 1
5 cores
VIOS 2
5 cores
IBM Power E880 IBM Power E880 IBM Power E880
VIOS 1
5 cores
VIOS 2
5 coresVIOS 1
5 cores
VIOS 2
5 cores
...
Infiniband
Switch
Infiniband
Switch
IBM Elastic Storage Server IBM Elastic Storage Server
Network
Switch (10 Gbit)
Network
Switch (10 Gbit)
VIOS 1
5 cores
VIOS 2
5 cores
Infiniband
10 Gbit
Ethernet
24
Solution Design II for IBM Power Servers
SAP Business Warehouse (SAP BW): Scale-out across multiple E870 servers
• 5 LPARs on 5 E870 running SAP BW powered by SAP HANA
- 4 TB memory each
- a core to memory ratio of 64 GB per core
- Operating System (initial) is SLES 11 SP4
- Linux boot disks are SAN attached (via VIOS and vFC)
16 TByte SAP BW: 4 worker nodes, 1 standby node, distributed across 5 E870 systems
• Each E870 is equipped with
- SR-IOV capable network adapters,
- Infiniband adapters
- FibreChannel adapters
- two VIOS LPARs
Filesystem for SAP HANA is GPFS
• Storage Servers are two ESS GL6
• Connectivity between SAP HANA (Spectrum Scale Client) and Storage Server (Spectrum Scale Server)
is preferable achieved via Infiniband 25
IBM Power E870 (70 cores / > 4 TByte)
Fill-up E870 with SAP BW Prod-Systems (Alternative TWO)
(Core-to-memory-ratio: 1 core per 64 GByte Memory)
VIOS 1
3 cores
VIOS 2
3 cores
64 GByte/ core (max. 40 cores, max. 4 TB)
100% 62,5%
E870 Building Block (max. 2)
50%80% 4 TB
SAP HANA(64 cores / 4 TB)
26
IBM Power E870
70 cores / > 4 TByte
HANA 1Host AutoFailover
4 worker
VIOS 1
3 cores
VIOS 2
3 cores
E870 configuration - 70 cores / 4 TByte of Memory per E870
27
IBM Power E870
70 cores / > 4 TByte
VIOS 1
3 cores
VIOS 2
3 cores
IBM Power E870
70 cores / > 4 TByte
VIOS 1
3 cores
VIOS 2
3 cores
IBM Power E870
70 cores / > 4 TByte
VIOS 1
3 cores
VIOS 2
3 cores
IBM Power E870
70 cores / > 4 TByte
VIOS 1
3 cores
VIOS 2
3 cores
IBM Power E870 IBM Power E870 IBM Power E870
VIOS 1
5 cores
VIOS 2
5 coresVIOS 1
5 cores
VIOS 2
5 cores
...
Infiniband
Switch
Infiniband
Switch
IBM Elastic Storage Server IBM Elastic Storage Server
Network
Switch (10 Gbit)
Network
Switch (10 Gbit)
VIOS 1
5 cores
VIOS 2
5 cores
Infiniband
10 Gbit
Ethernet
28
Elastic Storage Server Overview
Not just Spectrum Scale (aka Elastic Storage or GPFS)
Elastic Storage Server solutions consist of:
29
Power Systems
S822L
DCS3700
Expansions
EXP24
Expansions
ORESS
Spectrum Scale
aka:
Elastic Storage
IBM Spectrum Scale
Native RAID
Elastic Storage Server Offerings – GL Models
30
32
Model GL4
Analytics and Cloud4 Enclosures, 20U
232 NL-SAS, 2 SSD
13+ GB/Sec
Model GL6
PetaScale Storage6 Enclosures, 28U
348 NL-SAS, 2 SSD
25+ GB/sec
Model GL2
Analytics Focused2 Enclosures, 12U
116 NL-SAS, 2 SSD
6+ GB/Sec
5146 Machine
Type
•Power S822L Servers
•20 Cores Each
•1818-80E DCS3700 Expansion Chassis
•Red Hat 7
•Graphical User Interface
•Management Server and HMC
•Spectrum Scale Software
•Spectrum Scale Native RAID
•xCat or Platform Cluster Mgr. Opt.
•10 Gbps, 40 Gbps Enet, FDR Inifiniband
•From 116 to 348 Spinning Disk
•3 Years Hardware / 3 Year SW Maintenance
•Building Block approach to Growth
•High Capacity Storage for Analytics and Cloud Serving
•Uses 4U, 60 Drive Storage Enclosures
•2TB, 4TB or 6TB Drives
•A Client-Ready Petabyte in Single Rack!
34Page
Component Model for ICO
Use C
ases
Cre
ate
/Dele
te S
AP
Hana S
yste
m
Hardware
Orchestration
Virtualization
Power
(E880)
Guest: Suse
Host: PowerVM
IBM Cloud Orchestrator
MonitoringVirtualization-
management
Automation &
WorkflowsPatterns
OpenStack
Self-Service
GPFS
(ESS)
Operating System Linux (SLES)
Mid-Level Mgmt PowerVC HMC
Netzwerk
(Infiniband)
Leveraging Power Systems virtualization to provide superior management and
optimization
PowerVC Virtual Systems Management for PowerVM and PowerKVM
Image Deployment and Capture
Policy based VM Placement
VM Monitoring, Management, Mobility
Key Infrastructure as a Service (IaaS) elements
required for Cloud…
Integrated Server, Storage and Network Provisioning
and Mobility
Power Server
PowerVM
VM VM VM VM
Power Server
PowerVM
VM VM VM VM
Power Server
PowerKVM
VM VM VM VM
Power Server
PowerKVM
VM VM VM VM
VM Mobility
Optimized for:
• Availability
• Performance
• Energy
Managing a pool of resources with single system simplicity
VM Resilience and High Availability
On-Going Optimizations and Rebalancing
Security including Isolation and Multi-Tenancy
Differentiated with deep integration with IBM
Power Systems…
Vertically integrated and workload aware…
Horizontally integrated across server, storage and
networking…
VM Mobility
Use Case: Modify SAP HANA Production (optional)
37
Use Case: Modify SAP HANA non-Production (optional)
38
Migration to HANA on POWER
SAP heterogeneous system
copy procedures apply
Standard migrations using
Software Provisioning
Manager (SWPM) and
customer R3load exports
Software Upgrade Manager
with Database Migration
Option (SUM/DMO)
• can be used to combine a
release upgrade and the
heterogeneous database
migration to a new target system
39
The SAP HANA private Cloud @ client with Spectrum Protect (former TSM) Backup Integration (as is)
40
Enterprise server E880
“SAP HANA
Compute Cloud” IBM ESS
“SAP HANA
Storage Cloud”
ESS Model
GL6
PROD Site Backup Site
SAP Application Server
BW 7.3.1 / ERP 6.0
AIX 7.1
ABAP
TSM B/A Client
HANA DB Server
HANA Rev. 122
SLES 11 SP4
TSM B/A
Client
TSM Agent for HANA
Recovery Scenarios:
• Storage failure: no mirror yet available
• IBM Power failure: Live Partition Mobility on
• PROD Site
• Logical database failure: Restore from DB
• Backup
• Disaster Recovery with offsite tapes
SAN /
LANDisk
pool
Spectrum Protect
Server
Inhouse vault Remote vault
Next steps:
41
• HA and DR setup in two different data center
• Integration of SAP System Replication for critical SAP ERP
systems
• Setup of storage replication with spectrum scale
• Integration of a snapshot solution for SAP HANA
The SAP HANA private Cloud @ client with Spectrum Protect Backup Integration (scheduled for Q4/2016)
42
PROD SiteBackup Site
SAN /
LAN
Disk
pool
Spectrum Protect
Server
Inhouse vault
Enterprise server E880
“SAP HANA
Compute Cloud” IBM ESS
“SAP HANA
Storage Cloud”
ESS Model GL6
SAP Application
Server
BW 7.3.1 / ERP
6.0
AIX 7.1
ABAP
TSM B/A Client
HANA DB
Server
HANA Rev. 122
SLES 11 SP4
TSM B/A
Client
TSM Agent for
HANA
Disk
pool
Spectrum Protect
Server
Inhouse vault
Enterprise server E880
“SAP HANA
Compute Cloud”
ESS Model GL6
IBM ESS
“SAP HANA
Storage Cloud”
SAP Application
Server
BW 7.3.1 / ERP 6.0
AIX 7.1
ABAP
TSM B/A Client
HANA DB Server
HANA Rev. 122
SLES 11 SP4
TSM B/A Client
TSM Agent for HANARecovery Scenarios:
• Storage failure: SAP HANA SSR
• IBM Power failure: Live Partition Mobility on / to
• PROD / Backup Site
• Logical database failure: Restore from DB Backup
• Disaster Recovery with offsite tapes
Thank you for your attention !
Time for questions …