sap hana on cisco ucs: installation options

© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. of 182

White Paper

SAP HANA on Cisco UCS: Installation Options

Cisco UCS Configuration Principles for SAP HANA with Shared Storage and Shared

Network

July 2016


Contents

Introduction .............................................................................................................................................................. 4

Global Hardware Requirements for SAP HANA .................................................................................................... 4 CPU and Memory ................................................................................................................................................. 4 Network ................................................................................................................................................................. 6 Storage ................................................................................................................................................................. 8 Operating System ................................................................................................................................................. 9 High Availability ..................................................................................................................................................... 9

Solution Design Requirements ............................................................................................................................... 9 Solution for a Single SAP HANA System on a Single Server: Scale-Up (Bare Metal or Virtualized) .................. 10 Solution for a Single SAP HANA System on Multiple Servers: Scale-Out .......................................................... 10 Solution for Multiple SAP HANA Systems: Scale-Up (Bare Metal or Virtualized) ................................................ 10 Solution for Multiple SAP HANA Systems: Scale-Out (Bare Metal Only) ............................................................ 11 SAP HANA Cloud ............................................................................................................................................... 11 Private Cloud ...................................................................................................................................................... 11 Public Cloud ........................................................................................................................................................ 12 SAP HANA Hosting ............................................................................................................................................. 12

Cisco Solution for SAP HANA: Global Configuration Settings ......................................................................... 13 Cisco UCS Configuration .................................................................................................................................... 13 Chassis Connection Options ............................................................................................................................... 13 Chassis Connection in Pinning Mode ................................................................................................................. 13 Power Policy ....................................................................................................................................................... 14 Power-Control Policy .......................................................................................................................................... 15 BIOS Policy ......................................................................................................................................................... 15 Serial-over-LAN Policy ........................................................................................................................................ 19 Maintenance Policy ............................................................................................................................................. 19 Intelligent Platform Management Interface Access Profile .................................................................................. 19

Intel Ivy Bridge and Haswell–Based Cisco Solution for SAP HANA Scale-Out Design ................................... 20 Solution Components .......................................................................................................................................... 20 Cisco UCS Configuration .................................................................................................................................... 22 Server-Pool Policy Qualification .......................................................................................................................... 22 Server-Pool Policy .............................................................................................................................................. 25 Adapter Policy Configuration ............................................................................................................................... 25 Network Configuration ........................................................................................................................................ 27 Network-Control Policy ....................................................................................................................................... 29 LAN Configuration ............................................................................................................................................... 29 SAN Access Configuration .................................................................................................................................. 31 Service Profile Template Configuration ............................................................................................................... 32 Service Profile Deployment ................................................................................................................................. 37

Cisco Solution for SAP HANA Tailored Datacenter Integration ........................................................................ 39 Chassis Discovery Policy .................................................................................................................................... 39 Chassis Connection in Port-Channel Mode ........................................................................................................ 40 Pinning Option with Eight Uplinks ....................................................................................................................... 42 Storage Connectivity Options .............................................................................................................................. 43 Fibre Channel–over–Ethernet Storage Options .................................................................................................. 60 NFS Storage Options .......................................................................................................................................... 70

Boot Options .......................................................................................................................................................... 93 PXE Boot ............................................................................................................................................................ 93 SAN Boot ............................................................................................................................................................ 94 Local-Disk Boot ................................................................................................................................................... 96

Operating System Installation ............................................................................................................................ 101 SUSE Linux Enterprise Server .......................................................................................................................... 101 Red Hat Enterprise Linux Installation ................................................................................................................ 131


Storage Access for SAP HANA .......................................................................................................................... 141 Block Storage for SAP HANA Data and Log Files ............................................................................................ 141 File Storage for SAP HANA Data and Log Files ............................................................................................... 145 Block Storage for SAP HANA Shared File System ........................................................................................... 148

Cisco UCS Solution for SAP HANA TDI Shared Network ................................................................................. 149 Multiple SAP HANA System IDs in One Appliance ........................................................................................... 149 SAP HANA and SAP Application Server in One Appliance .............................................................................. 151 SAP HANA in an Existing Cisco UCS Deployment ........................................................................................... 152

Conclusion ........................................................................................................................................................... 154

For More Information ........................................................................................................................................... 154

Appendix A: Intel Westmere–Based Cisco Solution for SAP HANA Scale-Out ............................................. 155 Cisco UCS Configuration .................................................................................................................................. 156 Server-Pool Policy Qualification ........................................................................................................................ 157 Network Configuration ...................................................................................................................................... 160 LAN Tab Configuration ..................................................................................................................................... 161 SAN Access Configuration ................................................................................................................................ 164 Service Profile Template Configuration ............................................................................................................. 164 Service Profile Deployment ............................................................................................................................... 170

Appendix B: Direct-Attached NFS Failure Scenarios ....................................................................................... 174 Network-Attached Storage and Cisco UCS Appliance Ports ............................................................................ 174 Failure Scenario 1: Cisco UCS Fabric Interconnect Fails (Simple Failure Case) .............................................. 175 Failure Scenario 2: Cisco UCS Failure of Chassis IOM or All Uplinks to Fabric Interconnect ........................... 177 Failure Scenario 3: Underlying Multimode VIF Failure (Appliance Port) ........................................................... 178 Failure Scenario 4: Last Uplink on Cisco UCS Fabric Interconnect Fails .......................................................... 180 Failure Scenario 5: NetApp Controller Failure ................................................................................................... 181 Summary of Failure-Recovery Principles .......................................................................................................... 182


Introduction

This document describes the implementation of the SAP HANA Tailored Datacenter Integration (TDI) option on the

Cisco Unified Computing System™ (Cisco UCS®). This document is not a step-by-step installation and

configuration guide for this solution. It instead presents configuration principles, with detailed configuration settings

based on the individual situation.

SAP HANA is SAP SE’s implementation of in-memory database technology.

The SAP HANA database takes advantage of the low cost of main memory (RAM), the data-processing capabilities

of multicore processors, and the fast data access of solid-state disk (SSD) drives relative to traditional hard drives

to deliver better performance for analytical and transactional applications. It offers a multiple-engine query

processing environment that allows it to support relational data (with both row- and column-oriented physical

representations in a hybrid engine) as well as graph and text processing for semistructured and unstructured data

management within the same system. The SAP HANA database is 100 percent compliant with atomicity,

consistency, isolation, and durability (ACID) requirements.

For more information about SAP HANA, go to the SAP help portal: http://help.sap.com/hana/.

Global Hardware Requirements for SAP HANA

SAP Societas Europaea (SE) specifies hardware and software requirements for running SAP HANA systems.

Follow the guidelines provided in the SAP documentation at http://saphana.com for SAP HANA appliances and the

TDI option. SAP SE can change these requirements, so check the documentation regularly.

A list of certified servers for SAP HANA is published at http://global.sap.com/community/ebook/2014-09-02-hana-

hardware/enEN/index.html. The solution design discussed in this document is not based on a specific server type.

All Cisco UCS servers listed there can be used in this reference architecture. Server models introduced after the

publication of this document may also be supported. The servers in the “Supported Entry-Level Systems” section of

the list are for SAP HANA scale-up configurations with the SAP HANA TDI option only.

CPU and Memory

SAP HANA supports only specific CPU and memory configurations.

CPU

SAP HANA supports servers equipped with Intel Haswell EX architecture (Intel Xeon processor E7-8880v3, E7-

8890v3, or E7-8880Lv3 CPUs) or Intel Ivy Bridge EX architecture (Intel Xeon processor E7-2890v2, E7-4890v2, or

E7-8890v2 CPUs). Another option is to use Intel Xeon processor E5-26xx v2 or v3 or v4 for scale-up systems with

the SAP HANA TDI option.

Memory

The main-memory configurations supported are limited. SAP HANA is supported only if the memory configuration

follows these rules:

● Homogenous symmetric assembly of dual in-line memory modules (DIMMs): that is, no mixture of DIMM

size or speed

● Maximum use of all available memory channels

http://help.sap.com/hana/

http://global.sap.com/community/ebook/2014-09-02-hana-hardware/enEN/index.html

http://global.sap.com/community/ebook/2014-09-02-hana-hardware/enEN/index.html


● Memory per socket of from 64 GB up to 384 GB for a 2-socket system and up to 512 GB for a 4 or 8-socket

system for SAP NetWeaver Business Warehouse (BW) and DataMart

● Memory per socket of from 64 GB up to 768 GB for SAP Business Suite on SAP HANA (SoH) on 2-, 4-, or

8-socket server

● Maximum supported memory per socket for SoH on an 8-socket server

CPU and Memory Combinations

On the basis of the requirements listed in this document, servers for SAP HANA must be configured as shown in

Tables 1 and 2.

Table 1. SAP HANA Data Mart and Business Warehouse on SAP HANA (BWoH)–Optimized Hardware Configurations

SAP HANA Data Mart and Business Warehouse on SAP HANA (BWoH)–Optimized Hardware Configurations

Preconfigured System Sizes

Systems requirements for the listed system sizes:

● DIMM setting: homogenous symmetric assembly of DIMMs and maximum utilization of all 8 DDR memory channels per processor

● Supported CPU’s are Haswell EX 8880 v3, 8890 v3, or new 8880Lv3

● Storage requirements (the storage media itself needs to be protected from failures):

◦ Log volume ½ of the RAM size for systems ≤256 GB RAM and min. ½ TB for all other systems

◦ Data volume 3x RAM

◦ HANA share for scale-up systems: 1 x RAM

◦ HANA share for scale-up systems: 1 x RAM of a worker node for each 4 nodes in the cluster**

Single Node The number of cores are the minimal numbers!

128GB – 768GB 2x 18 cores Intel HSW CPU server systems

128GB – 2TB 4x 18 cores Intel HSW CPU server systems and SAP HANA SPS 11+ (older SAP HANA releases are supported with up to 1.5 TB)

128GB – 4TB 8x 18 cores Intel HSW CPU server systems and SAP HANA SPS 11+ (older SAP HANA releases are supported with up to 3 TB)

Scale Out Fully non-blocking network, up to 16 nodes through standard certification

512GB*, 1TB*, 1.5, 2 TB 4x 18 cores Intel HSW CPU server systems

1TB*, 2TB*, 3, 4 TB 8x 18 cores Intel HSW CPU server systems

• expansion for existing scale out installations (e.g. Ivybridge based systems) or Haswell scale out options below max. supported RAM capacity. ** For scale-out SAP HANA systems, the recommended disk space for /hana/shared/<sid> depends on the number of worker nodes. Per each four worker nodes of a given scale-out system, a disk space of 1x RAM of one worker node: Examples:

● 3+1 systems, 512 GB per node => Size installation = 1x 512 GB = 512 GB

● 4+1 systems, 512 GB per node => Size installation = 1x 512 GB = 512 GB

● 5+1 systems, 512 GB per node => Size installation = 2x 512 GB = 1 TB

● 6+1 systems, 512 GB per node => Size installation = 2x 512 GB = 1 TB

● 9+1 systems, 512 GB per node => Size installation = 3x 512 GB = 1.5 TB

Source: “SAP HANA Hardware Platform Based on Haswell EX,” SAP AG, 2015

Table 2. SAP HANA SoH-Optimized Hardware Configurations

SAP HANA SoH-Optimized Hardware Configurations

Preconfigured System Sizes

Systems requirements for the listed system sizes:

● DIMM setting: homogenous symmetric assembly of DIMMs and maximum utilization of all 8 DDR memory channels per processor

● Supported CPU’s are Haswell EX 8880 v3, 8890 v3, or new 8880Lv3

● Storage requirements (the storage media itself needs to be protected from failures):

◦ Log volume ½ of the RAM size for systems ≤256 GB RAM and min. ½ TB for all other systems

◦ Data volume 3x RAM


SAP HANA SoH-Optimized Hardware Configurations

◦ HANA share 1 x RAM

Single Node The number of cores are the minimal numbers! Single Node The number of cores are the minimal numbers!

128GB – 1.5TB 2x 18 cores Intel HSW CPU server systems 128GB – 3TB 4x 18 cores Intel HSW CPU server systems




128GB – 15TB 20x 18 cores Intel HSW CPU server systems

Scale Out* Fully non-blocking network, up to 4 nodes through standard certification

6TB 8x 18 cores Intel HSW CPU server systems

12TB 16x 18 cores Intel HSW CPU server systems

• SoH scale out HW configuration only recommended if 6 TB sizing is exceeded. Source: “SAP HANA Hardware Platform Based on Haswell EX,” SAP AG, 2015

Network

A SAP HANA data center deployment can range from a database running on a single host to a complex distributed

system with multiple hosts located at a primary site and one or more secondary sites and supporting a distributed

multiterabyte database with full fault and disaster recovery.

SAP HANA has several types of network communication channels to support the various SAP HANA scenarios

and configurations:

● Client zone: Channels used for external access to SAP HANA functions by end-user clients, administration

clients, and application servers, and for data provisioning through SQL or HTTP

● Internal zone: Channels used for SAP HANA internal communication within the database or, in a

distributed scenario, for communication between hosts

● Storage zone: Channels used for storage access (data persistence) and for backup and restore

procedures


Figure 1. An Overview of a SAP HANA Network

Details about the network requirements for SAP HANA are available in the SAP white paper at

http://scn.sap.com/docs/DOC-63221.

Table 3 provides a list of networks defined by SAP or Cisco or requested by customers.

Table 3. SAP HANA Networks

Name Use Case Solutions Bandwidth Requirements

Client Zone Networks

Application server network Communication between SAP application server and database

All 1 or 10 Gigabit Ethernet

Client network Communication between user or client application and database

All 1 or 10 Gigabit Ethernet

Data source network Data import and external data integration Optional for all SAP HANA systems

1 or 10 Gigabit Ethernet

Internal Zone Networks

Internode network Node-to-node communication within a scale-out configuration

Scale-out 10 Gigabit Ethernet

System replication network SAP HANA system replication For SAP HANA disaster tolerance

To be determined with customer

Storage Zone Networks

Backup network Data backup Optional for all SAP HANA systems

10 Gigabit Ethernet or 8-Gbps Fibre Channel

Storage network Communication between nodes and storage Scale-out storage TDI

10 Gigabit Ethernet or 8-Gbps Fibre Channel

Infrastructure-Related Networks

Administration network Infrastructure and SAP HANA administration Optional for all SAP HANA systems

1 Gigabit Ethernet

Boot network OS boot using Preboot Execution Environment (PXE) and Network File System (NFS) or Fibre Channel over Ethernet (FCoE)

Optional for all SAP HANA systems

1 Gigabit Ethernet

All networks need to be properly segmented and can be connected to the same core or backbone switch. The

customer’s high-availability requirements determine whether and how redundancy is applied for the various SAP

HANA network segments.

http://scn.sap.com/docs/DOC-63221


Note: Network security and segmentation is a function of the network switch and must be configured according

to the specifications of the switch vendor.

Based on the listed network requirements, every server must be equipped with two 1 or 10 Gigabit Ethernet

connectivity (10 Gigabit Ethernet is recommended) for scale-up systems to establish communication with the

application or user (client zone). If the storage for SAP HANA is external and accessed through the network, two

additional 10 Gigabit Ethernet interfaces are required for the storage zone.

For scale-out solutions, an additional redundant network for the communication between the SAP HANA nodes

with at least 10 Gigabit Ethernet connectivity is required (internal zone).

Storage

The storage used must be listed as part of an appliance in the SAP HANA Product Availability Matrix (PAM) or at

http://scn.sap.com/docs/DOC-52522. The storage is listed in the detailed description of a certified solution. Click

the entry in the Model column to see the details. The storage can also be listed as certified enterprise storage for

the SAP HANA TDI option at the same URL on the Certified Enterprise Storage tab.

All relevant information about storage requirements is documented in the white paper “SAP HANA Storage

Requirements” and is available at http://scn.sap.com/docs/DOC-48516.

Note: SAP does not differentiate between a virtualized SAP HANA system with 64 GB of memory and a physical

SAP HANA system with 2 terabytes (TB) memory in terms of storage requirements. If the storage is certified and

has, for example, four nodes, you can run four virtual machines or four bare-metal systems, or a mixture of the two,

on this storage device.

Using the same storage for multiple applications can significantly degrade performance of the SAP HANA systems

and the other applications. You therefore should separate the storage for SAP HANA systems from the storage for

other applications. You must specify the storage options with your storage vendor based on your requirements.

File System Sizes

To install and operate SAP HANA, specify the file system layout and sizes as shown in Figure 2.




Figure 2.

Note: The recommended file system layout for a SAP HANA TDI option differs from hat shown in Figure 2. For

/hana/data, the minimum size is 1 x RAM instead of 3 x RAM, with 1.5 x RAM recommended. Also, for solutions

based on Intel Xeon processor E7-x890v2 CPU, the size of the log volume differs from Figure 2. The log size is

one-half of the server memory size for systems with 256 GB of memory or less, and a minimum of 512 GB for

systems with 512 GB of memory or more.

Operating System

For information about the supported operating systems for SAP HANA, see SAP Note 2235581: SAP HANA:

Supported Operating Systems.

High Availability

A SAP HANA scale-out system has an integrated high-availability function. If the SAP HANA system is configured

with a standby node, a failed part of SAP HANA will start on the standby node automatically. The infrastructure for

a SAP HANA scale-out solution must have no single point of failure. After a component failure, the SAP HANA

database must still be operational and running. For automatic host failover, the proper implementation and

operation of the SAP HANA storage connector API is required.

Hardware and software components should include the following:

● Internal storage: A RAID-based configuration is preferred.

● External storage: Redundant data paths, dual controllers, and a RAID-based configuration are preferred.

● Ethernet switches: Two or more independent switches should be used.

For the latest information from SAP, see http://saphana.com/ or http://service.sap.com/notes.

Solution Design Requirements

Because multiple implementation options are available, you need to define your requirements before you can

select the solution components. This section defines the basic requirements for each option.

http://saphana.com/

http://service.sap.com/notes


Solution for a Single SAP HANA System on a Single Server: Scale-Up (Bare Metal or Virtualized)

A scale-up solution is, for some SAP HANA use cases, the only fully supported option. In general, this solution

provides the best SAP HANA performance. All data and processes are located on the same server and can be

accessed locally, and no network communication with other SAP HANA nodes is required. Nearly all certified SAP

HANA scale-up solutions are based on a standalone rack-mount server and use internal storage.

The network requirements for this option depend on the client and application access and storage connections. If

you don’t need system replication or a backup network, a 1 Gigabit Ethernet (access) network and a 10 Gigabit

Ethernet or 8-Gbps Fibre Channel (storage) network are sufficient to run a SAP HANA scale-up solution.

The storage requirements for this option depend on I/O performance. You should consider storage with low latency

(less than 1 microsecond) and a high write rate. If you use shared storage, you need to keep in mind that, from a

storage perspective, SAP HANA is a write-only application, whereas most other applications are read intensive.

For a virtualized SAP HANA installation, remember that hardware components, such as network interfaces and

host bus adapters (HBAs), are shared. If possible, use dedicated network adapters and HBAs for each virtualized

SAP HANA system.

Solution for a Single SAP HANA System on Multiple Servers: Scale-Out

You should use a scale-out solution if the SAP HANA system does not fit into the main memory of a single server

based on the rules defined by SAP. Some use cases are either not supported or not fully supported on scale-out

solutions. Check with SAP to determine whether your use case can be deployed on a scale-out solution.

The network requirements for this option are higher than for scale-up systems. In addition to the client and

application access and storage access networks, you also must have a node-to-node network. If you don’t need

system replication or a backup network, a 10 Gigabit Ethernet (access) network and a 10 Gigabit Ethernet (node-

to-node) and a 10 Gigabit Ethernet or 8-Gbps Fibre Channel (storage) network are required to run SAP HANA in a

scale-out configuration.

The storage requirements for this option depend on throughput performance. You should consider a storage

system with approximately 5 to 8 Gbps per server. If you use shared storage, you should keep in mind that SAP

HANA is, from a storage perspective, a write-only application, whereas most other applications are read intensive.

A virtualized SAP HANA scale-out solution is under controlled availability by SAP as of November 2015. Refer SAP

Note 2157587 for more details.

Solution for Multiple SAP HANA Systems: Scale-Up (Bare Metal or Virtualized)

Using the SAP HANA TDI option for shared storage and a shared network, you can build a solution to run multiple

SAP HANA scale-up systems on shared infrastructure. One approach is to use a SAP HANA scale-out solution

and install one SAP HANA system per server. A 4 + 1 scale-out solution (four active nodes and one standby node)

includes all the components needed to run five SAP HANA systems based on the TDI key performance indicators

(KPIs).

The network requirements are the same as for a single SAP system.

The storage requirements for this option are not easy to define. They depend mainly on the use case and

performance expectations.

Storage performance KPIs for a TDI implementation are much lower than storage performance KPIs for a scale-out

solution, and these KPIs are much lower than KPIs for a scale-up solution. For an analytics processing system,


storage performance is important for writing updates in the database, for starting the database, and for cases that

require recovery. All read queries are answered out of the main memory. For a transactional system, storage

performance significantly affects the overall SAP HANA performance. Every commit operation must be written to

the disk to make it persistent. If storage is slow, the commit rate is slow, and the end-user experience can be poor.

There is no single rule that guides storage sizing for this option.

For a virtualized SAP HANA installation, remember that some hardware components are shared: for example,

network interfaces and HBAs. If possible, you should use dedicated network adapters and HBAs for each

virtualized SAP HANA system.

Solution for Multiple SAP HANA Systems: Scale-Out (Bare Metal Only)

Using the SAP HANA TDI option for shared storage and a shared network, you can build a solution to run multiple

SAP HANA scale-out systems on shared infrastructure. One approach is to use a SAP HANA scale-out solution

and install two or more SAP HANA systems on it. You can use an 11 + 1 scale-out solution (11 active nodes and 1

standby node) to install two 5 + 1 systems or three 3 + 1 systems or any other supported scale-out configuration.

The network requirements are the same as for a single SAP HANA system.

Refer to the storage part of the section “Solution for Multiple SAP HANA Systems: Scale-Up (Bare Metal or

Virtualized)” earlier in this document.

A virtualized SAP HANA scale-out solution is under controlled availability by SAP as of November 2015. Refer SAP

Note 2157587 for more details.

SAP HANA Cloud

One popular way to deploy SAP HANA is in the cloud. There are hundreds of interpretations of cloud computing.

The scope of this document is confined to four high-level cloud models that are relevant for SAP HANA:

● SAP HANA Enterprise Cloud, offered by SAP (this option is not addressed in this document)

● Private cloud, offered by the IT department to multiple customers within the same company

● Public cloud, in which SAP HANA is offered through a website, based on a global service description,

through which it can be accessed with few mouse clicks and the user’s credit card information

● SAP HANA hosting, in which a service provider runs the SAP HANA system based on an individual contract

that includes a service-level agreement (SLA) that specifies availability, performance, and security

The cloud model used affects the infrastructure setup and configuration. Following are some examples:

● The performance requirements define the number of SAP HANA systems that can run on the same

hypervisor, storage system, or network.

● The security requirements influence whether or not dedicated hardware is required: that is, whether or not

you must use a dedicated physical host, storage system, or network equipment.

● The availability requirements influence the way that SAP HANA is deployed in the data center, the

requirements for infrastructure and application clustering, and the data-replication technology and backup

and restore procedures.

Private Cloud

The private cloud model for SAP HANA represents a broad set of options. You can use the scale-up solution for

multiple SAP HANA systems (bare metal or virtualized) with additional automation tools to provide a flexible and

easy way to provision and deprovision SAP HANA systems. You can also use existing infrastructure for virtual


desktop infrastructure (VDI) or other traditional applications and add SAP HANA to it. In most private cloud

deployments, SAP HANA is not the only application that runs on the shared infrastructure.

The network requirements for SAP HANA in a private cloud are the same as for multiple SAP systems. You also

can consider a basic implementation of security and network traffic separation.

The storage requirements are not easy to define. You must consider the performance requirements for SAP HANA

and the other applications, as described in the section “Solution for Multiple SAP HANA Systems: Scale-Up (Bare

Metal or Virtualized)” earlier in this document. You also need to consider requirements for backup location, data

mobility, use of multiple data centers, etc. Storage also should follow the data center best practices that are already

in place.


network interfaces and HBAs. If possible, you should use dedicated network adapters and HBAs for each

virtualized SAP HANA system.

Public Cloud

A public-cloud solution is used to run many small to medium-size SAP HANA systems. Most of those systems are

virtualized, and the solution is 100 percent automated. A self-service portal for the end customer is used to request,

deploy, and operate the SAP HANA system. There are no individual SLAs; you have to accept the predefined

agreement to close the contract.

The network requirements for SAP HANA in a private cloud are the same as for multiple SAP systems. In addition,

you must develop a security design and implement network traffic separation.

The storage requirements depend on the flexibility and manageability of the storage subsystems. The performance

characteristics must be good enough to run SAP HANA and the other applications. The performance requirements

for SAP HANA are described in the section “Solution for Multiple SAP HANA Systems: Scale-Up (Bare Metal or

Virtualized)” Also important are the requirements for backup location, data mobility, multiple data centers, etc.

Storage also should follow the data center best practices that are already in place.


network interfaces and HBAs. If possible you should dedicate network adapters and HBAs for each virtualized SAP

HANA system.

SAP HANA Hosting

A SAP HANA hosting solution is used to run small to large-size SAP HANA systems. Some of these systems may

be virtualized, some may be bare metal scale-up systems, and some may be scale-out systems. The solution is

automated to a level that makes sense for the owner. A self-service portal for the end customer is not mandatory,

but can be used. Individual contracts are created, with a dedicated SLA per end customer. The infrastructure must

be able to provide flexibility based on the individual performance, security, and availability requirements of the end

customers.

The network requirements for a SAP HANA hosting solution are the same as for multiple SAP systems. In addition,

security and network traffic separation are required.

The storage configuration should be defined in a way to provide the best performance. The performance

requirements for SAP HANA are described in the section “Solution for Multiple SAP HANA Systems: Scale-Up

(Bare Metal or Virtualized)” earlier in this document. The other storage-related features and functions should follow

the data center best practices that are already in place.



network interfaces and HBAs. If possible, you should dedicate network adapters and HBAs for each virtualized

SAP HANA system.

Cisco Solution for SAP HANA: Global Configuration Settings

This section describes the global configuration settings for the Cisco solution for SAP HANA.

Cisco UCS Configuration

This document does not present all the Cisco UCS configuration options. For example, this document does not

describe how to configure administrative network access or how to configure MAC address, World Wide Name

(WWN) address, or IP address pools. The focus of this document is on configuring the system to run SAP HANA.

Some specific Cisco UCS configuration options are required to meet SAP’s requirements for SAP HANA.

Chassis Connection Options

For Cisco UCS 2200 Series Fabric Extenders, two configuration options are available: pinning and Port Channels.

Because every SAP HANA node will communicate with every other SAP HANA node through multiple I/O streams,

the Port Channel option provides better features. However, for use cases in which only a few large I/O streams are

used, the pinning option provides more stable performance. Because communication behavior can differ from use

case to use case, SAP defined a single-stream network performance test as part of the hardware validation tool

(hwcct or hwval). To pass the hwval test, the pinning mode was used for the Cisco solutions.

Chassis Connection in Pinning Mode

In the pinning mode, every virtual interface card (VIC) in a Cisco UCS B-Series Blade Server is pinned to an uplink

cable from the fabric extender (or I/O module [IOM]) to the fabric interconnect based on the available number of

uplinks. In most cases, the chassis are connected with four 10 Gigabit Ethernet cables per IOM to the fabric

interconnect. The chassis backplane provides eight internal connections; a half-width blade can use one

connection, and a full-width blade can use two connections. Every connector is mapped to a VIC on the blade, and

every VIC is represented by a virtual network interface connection (vCON) in Cisco UCS Manager.

To run SAP HANA on an infrastructure with four uplinks per IOM, use Tables 4, 5, and 6 to understand the pinning

of IOM uplink ports (P1 to P4) and vCON. This pinning information is used when the virtual network interface card

(vNIC) and virtual host bus adapter (vHBA) placement policy is defined.

Table 4. Cisco UCS 5108 Chassis with Eight Half-Width Blades (Cisco UCS B230 M2 or B200 M4)

Port1: vCON1 Port2: vCON1



Port3: vCON1 Port4:- vCON1

Table 5. Cisco UCS 5108 Chassis with Four Full-Width Blades (Cisco UCS B440 M2 or B260 M4 Blade Server)






Table 6. Cisco UCS 5108 Chassis with Two Full-Width Double-High Blades (Cisco UCS B460 M4 Blade Server)





The best configuration found to pass the hwcct test uses pinning mode.

Choose Equipment > Global Policies, and for Chassis/FEX Discovery Policy, select None for Link Grouping

Preference to use the pinning mode (Figure 3).

Figure 3. Chassis Discovery Policy

In the pinning mode example in Figure 4, nothing is listed in the Port Channel column.

Figure 4. IOM Fabric Ports with Pinning Mode

In the port-channel example in Figure 5, the Port Channel column lists the port channel used.

Figure 5. IOM Fabric Ports with Port Channel Mode

Power Policy

To run Cisco UCS with two independent power-distribution units (PDUs), Redundancy must be configured as Grid

(Figure 6).


Figure 6. Power Policy

Power-Control Policy

The Cisco UCS power-capping feature is designed to save power in traditional data center use cases. This feature

does not fit with the high-performance behavior of SAP HANA. If power capping is configured on Cisco UCS

globally, the power-control policy for the SAP HANA nodes makes sure that the power capping does not apply to

the nodes. The Power Capping feature should be set to No Cap (Figure 7).

Figure 7. Power-Control Policy for SAP HANA Nodes

BIOS Policy

To get the best performance for SAP HANA, you must configure the server BIOS accordantly. On the Main tab, the

only change made was disabling Quiet Boot to show details in the posting (Figure 8).

Figure 8. BIOS Policy Main Settings


For SAP HANA, SAP also recommends disabling all processor C-states (Figure 9). This configuration forces the

CPU to stay on the maximum frequency and allows SAP HANA to run with the best performance.

Figure 9. BIOS Policy: Advanced > Processor

No changes are required on the Intel Directed IO tab (Figure 10).

Figure 10. BIOS Policy: Advanced > Intel Directed IO

On the RAS Memory tab, choose maximum-performance and enable NUMA (Figure 11), to allow non-uniform

memory access (NUMA).


Figure 11. BIOS Policy: Advanced > RAS Memory

On the Serial Port tab, be sure that Serial Port A is enabled (Figure 12).

Figure 12. BIOS Policy: Advanced > Serial Port

No changes are required on the USB tab (Figure 13).

Figure 13. BIOS Policy: Advanced > USB


No changes are required on the PCI Configuration tab (Figure 14).

Figure 14. BIOS Policy: Advanced > PCI Configuration

No changes are required on the Boot Options tab (Figure 15).

Figure 15. BIOS Policy: Boot Options

For Console Redirection, choose serial-port-a, set BAUD Rate to 115200, and enable the Legacy OS Redirect

option. These settings are used for serial console access over the LAN to all SAP HANA servers (Figure 16).

Figure 16. BIOS Policy: Server Management


Serial-over-LAN Policy

Serial-over-LAN (SoL) policy is required to enable console access to all SAP HANA servers through the Secure

Shell (SSH) Protocol from the management network. This policy is used when the server hangs or a Linux kernel

failure dump is required. Be sure that the configured speed is the same as that configured on the Server

Management tab for the BIOS policy (Figure 17).

Figure 17. Serial-over-LAN Policy

Maintenance Policy

Cisco recommends defining a maintenance policy with Reboot Policy set to User Ack for the SAP HANA server

(Figure 18). This policy helps ensure that a configuration change in Cisco UCS does not automatically force all

SAP HANA servers to reboot. The administrator has to acknowledge the reboot for the servers changed in Cisco

UCS; otherwise, the configuration change will take effect when the server reboots through an OS command.

Figure 18. Maintenance Policy

Intelligent Platform Management Interface Access Profile

Serial-over-LAN access requires intelligent platform management interface (IPMI) access to the board controller

(Figure 19). This profile is also used for the STONITH function of the SAP HANA mount API to shut down a

hanging server. The default user is sapadm with the password cisco.

Note: STONITH stands for “shoot the other node in the head.”


Figure 19. IPMI Access Profile

Intel Ivy Bridge and Haswell–Based Cisco Solution for SAP HANA Scale-Out Design

With the introduction of new-generation Intel Xeon E7 processors, the new servers for SAP HANA needed to be

certified. Also, SAP has improved many elements on the software stack, introduced new use cases, and defined

new implementation options. The requirement for a closed appliance-like solution was relaxed somewhat, and the

introduction and use of the SAP HANA TDI option required the whole solution design to be reconsidered. To meet

the new requirements, Cisco developed the second generation of solutions for SAP HANA. This new generation of

solutions enabled the use of all features of Cisco UCS introduced after Cisco UCS Manager Release 2.1, such as

directly connected storage, VLAN groups, and managed Cisco UCS C-Series Rack Servers.

Solution Components

The solution design that Cisco developed for SAP HANA uses the following components:

● Cisco UCS

◦ Two Cisco UCS 6248UP 48-Port or 6296UP 96-Port Fabric Interconnects

◦ Cisco UCS B260 or B460 M4 Blade Server option

◦ Two to 9 Cisco UCS 5108 Blade Chassis (if B260 or B460 is used)

◦ Two Cisco UCS 2204 Fabric Extenders connected with four 10 Gigabit Ethernet interfaces to the

fabric interconnect

◦ Three to 16 Cisco UCS B460 M4 High-Performance Blade Servers with two Cisco UCS VIC

1280 and two Cisco UCS VIC 1240 cards

◦ Cisco UCS C460 M4 Option


◦ Three to 16 Cisco UCS C460 M4 High-Performance Rack Servers with two Cisco UCS VIC 1225

cards

◦ Each server uses four 10 Gigabit Ethernet for dedicated bandwidth

● Cisco switches

◦ One Cisco Nexus 3524, Nexus 5548, or Nexus 9372 Switch for 10 Gigabit Ethernet connectivity between

the two Cisco UCS fabric interconnects (This switch is for fail-over purposes only; no data traffic through

this switch.)

● Storage

◦ One or two EMC VNX5400 unified storage systems, with NFS for the SAP HANA shared area and Fibre

Channel for the OS, SAP HANA data files, and log files

◦ One to three NetApp FAS8040 controllers with NFS for the OS, SAP HANA data files, and log files

● Cisco UCS C-Series Rack Servers (only if PXE boot is used)

◦ Two Cisco UCS C220 M3 or M4 Rack Servers to host the management virtual machines

◦ One SUSE Linux Enterprise Server (SLES) or Red Hat Enterprise Linux (RHEL)–based management

server as a VMware virtual machine

◦ One Microsoft Windows–based monitoring server as a VMware virtual machine

Note: The Cisco UCS C-Series servers and the management virtual machines are not part of the validation and

certification effort, and you do not have to implement the restrictions from SAP regarding changes. Every VMware

ESX version that is supported by the Cisco UCS C220 M3 or M4 can be installed. The Cisco UCS C220 M3 or M4

also can be integrated into a VMware vCenter domain. An NFS share can be mounted as a data store, and files

can be moved from internal storage to external storage.

In this design, Cisco UCS is directly connected to the data center network, and, therefore, you must configure the

correct VLAN IDs and IP address ranges on all components.

Figure 20 shows the Cisco Solution for SAP HANA Scale-Out.


Figure 20. Cisco Solution for SAP HANA Scale-Out


Some Cisco UCS configurations specific to the second-generation solution design are based on the Cisco UCS

B260 or B460 M4 Blade Server and Cisco UCS C460 M4 Rack Server.

Server-Pool Policy Qualification

The configuration of a server to run SAP HANA is well defined by SAP. Within Cisco UCS, you can specify a policy

to collect all servers for SAP HANA in a pool. The definition in Figure 21 specifies all servers with 1024 GB of

memory and 60 cores running at a frequency of 2800 MHz or higher for Ivy Bridge–based systems. For Haswell-

based systems, 1536 GB of memory and 2500 MHz are required.


Figure 21. Server-Pool Policy Qualification: Ivy Bridge Example

The processor architecture is not of interest because the combination of 60 cores with 2800 MHz or more applies

only to the Intel Xeon processor E7-4880v2 (Ivy Bridge CPU) 4-socket server (Figure 22). The combination of 72

cores and 2500 MHz or more applies to the Intel Xeon E7-8890 v3 (Haswell CPU) 4-socket server.

Figure 22. CPU Qualification Properties: Ivy Bridge Example


The capacity is defined as exactly 1024 GB of memory (Figure 23) for Ivy Bridge and 1536 GB of memory for

Haswell systems.

Figure 23. Memory Qualification Properties: Ivy Bridge Example


Server-Pool Policy

The server pool for the SAP HANA nodes and the qualification policy are also defined. In this case, the two

definitions are mapped together (Figure 24).

Figure 24. Server-Pool Policy: Ivy Bridge Example.

As a result, all servers with the specified qualification are now available in the server pool (Figure 25).

Figure 25. List of Servers in Server Pool: Ivy Bridge Example

Adapter Policy Configuration

Figure 26 shows newly created Ethernet adapter policy Linux-B460 with Receive-Side Scaling (RSS), Receive

Queues, and Interrupts values defined. You must use this policy for the SAP HANA internal network to provide the

best network performance with SLES 11 for both Ivy Bridge and Haswell solutions.


Figure 26. Adapter Policy Linux-B460 for Use with Cisco B260 M4 and B460 M4 Servers


Network Configuration

The core requirements from SAP for SAP HANA are met by Cisco UCS defaults. Cisco UCS is based on 10

Gigabit Ethernet and provides redundancy through the dual-fabric concept (Figure 27).

Figure 27. Network Paths with Solution Design

Each Cisco UCS chassis is linked through four 10 Gigabit Ethernet connections to each Cisco UCS fabric

interconnect. Those southbound connections can be configured in Port Channel mode or pinning mode. For the

preconfigured solution, the pinning mode was used for better control of the network traffic. The service profile

configuration helps ensure that during normal operation, all traffic in the internal zone is on fabric A, and all other

traffic (client-zone and storage-zone traffic) is on fabric B. The management traffic is also on fabric A. This

configuration helps ensure that the network traffic is distributed across both fabrics.

With this configuration, the internode traffic flows only from the blade to the fabric interconnect and back to the

blade. The NFS storage traffic for each SAP HANA system should be limited to one fabric, because if multiple SAP

HANA systems are running on the same infrastructure, the fabric can be different for every SAP HANA system.

Only network traffic that has to go to the data center network will leave Cisco UCS fabric interconnects.


Because the Cisco solution with EMC storage uses Fibre Channel for the data and log volumes, two HBAs also

must be configured: one per fabric. The multipath driver for SLES is used to provide path redundancy and to

distribute the traffic over both fabrics as required. With integrated algorithms for bandwidth allocation and quality of

service (QoS), Cisco UCS helps provide the best traffic distribution.

If an IOM or cable fails, internode and IP storage traffic for the network has to go through the internal network

switch (Cisco Nexus 3524 platform), as shown in Figure 28.

Figure 28. Network Traffic If IOM or Cable Fails


Network-Control Policy

A network-control policy is required to help ensure that only a warning is generated in response to an uplink failure

to the internal Cisco Nexus switch. This policy also must help ensure that the link doesn’t go down (Figure 29).

Figure 29. Network-Control Policy

LAN Configuration

In Cisco UCS, all network types for a SAP HANA system are defined in VLANs. Figure 30 shows a sample list of

networks based on the SAP HANA network requirements document. The VLAN IDs must be defined as required

for the individual installation; this list is only an example.

Figure 30. VLAN Definition in Cisco UCS (Example)

Each VLAN is mapped to a vNIC template to specify the characteristics of a specific network. The vNIC template

configuration includes settings such as maximum transmission unit (MTU) size, failover capabilities, and MAC

address pools.


In the Cisco solution for SAP HANA discussed here, up to nine vNICs must be defined (Figure 31).

Figure 31. vNIC Templates

The default setup is configured so that for most SAP HANA use cases the network traffic is well distributed across

the two fabrics (fabric A and fabric B). In special cases, this distribution may need to be rebalanced for better

overall performance. This rebalancing can be performed in the vNIC template with the Fabric ID setting. Be sure

that the MTU setting matches the configuration in your data center. For the internal vNIC, always set the MTU

value to 9000 for the best performance (Figure 32).

Figure 32. vNIC Template Details


For VLANs that are not allowed to go to the data center network, such as T01-internal or T01-storage, set the

network-control policy to link-up (Figure 33), as defined earlier in this document.

Figure 33. vNIC Template Details (Internal)

SAN Access Configuration

For SAN access, you must configure vHBAs. A best practice is to configure vHBA templates: one for fabric A

(Figure 34) and one for fabric B. If you want the Fibre Channel traffic for the OS, data, and log to be separate, one

vHBA per type and fabric is required.

Figure 34. Creating a vHBA Template


Service Profile Template Configuration

Now that all LAN and SAN access configurations and all policies relevant to SAP HANA are defined, a service

profile template can be configured.

On the General tab, select the universally unique ID (UUID), server pool, and maintenance policy (Figure 35).

Figure 35. Service Profile Template: General Tab

On the Storage tab, define the World Wide Node Name (WWNN) configuration and the vHBAs based on the vHBA

templates (Figure 36).

Figure 36. Service Profile Template: Storage Tab


On the Network tab, define the required vNICs based on the vNIC templates. To map the pinning at the IOM layer,

you must manually configure the vNIC and vHBA placement (Figure 37).

Figure 37. Service Profile Template: Network Tab

In the Actions pane, select Modify vNIC/vHBA Placement. For the service profile template used for all servers in

slot 3, assign all vNICs among the four vCONs as shown in the sample configuration (Figure 38).

Figure 38. vNIC and vHBA Placement for Slot-3 Servers

For the service profile template used for all servers in slot 7, assign all vNICs among the four vCONs as shown in

the sample configuration (Figure 39).


Figure 39. vNIC and vHBA Placement for Slot-7 Servers


On the Boot Order tab, use PXE boot for the Cisco solution (Figure 40).

Figure 40. Service Profile Template: Boot-Order Sample for PXE Boot and SAN Boot (Specific to EMC-Based Scale-Out Solution)

On the Policies tab, all the definitions already created are selected (Figures 41 and 42).


Figure 41. Service Profile Template: Policies Tab (Part 1)


In the Cisco solution, nothing is configured on the iSCSI vNICs tab for Small Computer System Interface over IP

(iSCSI) vNICs.


Service Profile Deployment

The basic settings are defined and captured in the service profile template for SAP HANA nodes. To deploy new

SAP HANA nodes, the only step that is required in Cisco UCS is to create service profiles from the specified

service profile template (Figure 43).

Figure 43. Service Profile Template: Before Service Profiles Are Created

Click Create Service Profile from Template, and a new window will appear. In this window, specify the service

profile name prefix, the starting number, and the number of service profiles that you want to create (Figure 44).

Figure 44. Creating a Service Profile from a Template

The specified number of service profiles will be deployed and mapped to a blade if one is physically available

(Figure 45).


Figure 45. Defined Service Profiles

The configuration details for every service profile now follow the definition that was created previously in the

service profile template. No intervention is required, and every service profile is ready for SAP HANA.


Cisco Solution for SAP HANA Tailored Datacenter Integration

When Cisco solution components are used for a SAP HANA installation, the Cisco UCS and Cisco Nexus devices

are preconfigured as discussed earlier in this document.

The SAP HANA TDI option allows more openness and variation in the configuration of the components, but you

still must use the certified components such as the Cisco UCS B460 M4 server and Cisco Nexus 3000 Series

switches. A critical point to consider is the connection between the fabric interconnects and the chassis. As

documented earlier, the Cisco solution comes with four 10 Gigabit Ethernet uplinks from the IOM to the fabric

interconnect, which provides two 20-Gbps uplinks for each Cisco UCS B460 M4 server. This bandwidth is suitable

for running a SAP HANA scale-out system. If additional bandwidth is needed, for example, for backup or data-

replication purposes, you can configure the solution differently.

Chassis Discovery Policy

The first step in implementing a Cisco solution for SAP HANA TDI is a software configuration change. Change the

chassis discovery policy from None to Port Channel (Figure 46).

Figure 46. Chassis Discovery Policy

Alternatively, if you are not using Cisco UCS exclusively for SAP HANA, you can change this configuration on a

per-chassis basis (Figure 47).

Figure 47. Chassis Connectivity Policy


This simple change can have a huge effect on the SAP HANA solution. The details of the pinning mode, which is

used if None is selected, are discussed in the section “Chassis Connection Options” earlier in this document.

Chassis Connection in Port-Channel Mode

Because SAP uses single-stream bidirectional network communication between SAP HANA nodes, a Port

Channel-based configuration of Cisco UCS will not always provide the best performance. With the implemented

hashing algorithm on the IOM and fabric interconnect, the outbound traffic (traffic from the IOM to the fabric

interconnect) as well as the inbound traffic (traffic from the fabric interconnect to the IOM) can travel over the same

cable. Therefore, in a unidirectional test, the results will be greater than or equal to 9.5 Gbps outbound, and greater

than or equal to 9.5 Gbps inbound. A bidirectional test has twice the outbound and twice the inbound traffic on the

same cable, and the results will be 4.5 Gbps or less per direction.

For a SAP HANA system, the Port Channel option provides some benefits because a single node can use eight 10

Gigabit Ethernet connections in burst mode to communicate with other nodes and the storage or the application

server, whereas the pinning mode limits communication to two 10 Gigabit Ethernet connections.

The Port Channel mode is preferred in cases in which Cisco UCS is used as a shared platform running SAP HANA

and other applications at the same time. In such cases, the following server distribution options are recommended:

● Use one server per chassis for SAP HANA and the others for applications other than SAP HANA.

● Use servers 1 and 7 per chassis for SAP HANA and the others for applications other than SAP HANA.

● Use all servers per chassis for SAP HANA, with each server used for a different system ID (SID).

● Use all servers per chassis for a single SAP HANA SID.

The use of Port Channel mode also simplifies the vNIC and service profile template configuration.

Instead of a specific vNIC and vHBA placement policy, you can use a global LAN connectivity policy. The LAN

connectivity policy is used to specify in a single place the type and order of all the vNICs that the SAP HANA server

will use (Figure 48).

Figure 48. LAN Connectivity Policy

The LAN connectivity policy configuration includes all vNICs that are used for SAP HANA. You do not have to use

all specified vNIC templates in all installations. Mandatory interfaces are Admin, Internal, AppServer, and Client. All

other interfaces are optional.

The vNICs are mapped to the vNIC template configured previously in this document (Figure 49).


Figure 49. Creating a vNIC for LAN Connectivity Policy

On the Network tab of the service profile template, select Modify vNIC/vHBA Placement and choose Let System

Perform Placement from the Select Placement menu (Figure 50).

Figure 50. Modifying vNIC and vHBA Placement for a Port Channel

Select the LAN connectivity policy HANA-Order. All configured vNICs are defined in the correct order and mapped

to the correct fabric (Figure 51).



Pinning Option with Eight Uplinks

The next step requires changes in the hardware. Instead of the Cisco UCS 2204 Fabric Extender IOM with four 10

Gigabit Ethernet ports, you use the Cisco UCS 2208 Fabric Extender IOM with eight 10 Gigabit Ethernet ports.

With this configuration, every vCON is pinned to a dedicated uplink port, and the bandwidth per server is four 10

Gigabit Ethernet uplinks (Tables 7, 8, and 9).

Table 7. Cisco UCS 5108 Chassis with Eight Half-Width Blades (Cisco UCS B200 M4)





Table 8. Cisco UCS 5108 Chassis with Four Full-Width Blades (Cisco UCS B440 M2 or B260 M4)






Table 9. Cisco UCS 5108 Chassis with Two Full-Width Double-High Blades (Cisco UCS B460 M4)





With eight uplinks, Cisco recommends that you use the same LAN connectivity policy as discussed earlier in the

sections “Chassis Connection in Port-Channel Mode” and “Chassis Connection Options.”

Storage Connectivity Options

The next step is to set the storage connectivity option that you will use for this installation. The most common

connectivity options are shown here. The boot option and the SAP HANA configuration depend on the selection

you make.

Fibre Channel Storage Options

Before you can use Fibre Channel storage, you must change the unified ports on the fabric interconnects from

Ethernet to Fibre Channel (Figure 52).

Figure 52. Fabric Interconnect: Configuring Unified Ports


If an expansion module is installed, a best practice is to use the fixed module ports as server ports or Ethernet

uplink ports (Figure 53).

Figure 53. Unified Ports: Configuring Fixed-Module Ports

In the expansion module, move the slider from the right to the left; the color for the ports changes to magenta

(Figure 54). After you click Finish, the expansion module reboots to reconfigure the ports.


Figure 54. Unified Ports: Configuring Expansion-Module Ports

VSAN Configuration

If the SAN uses VSAN or equivalent technology, you must configure Cisco UCS to match.

On the SAN tab, the VSAN configuration can be performed as a global configuration or as a fabric-based

configuration. In the example in Figure 55, a fabric-based configuration is defined, with VSAN 10 used for fabric A,

and VSAN 20 used for fabric B.

Figure 55. VSAN Configuration

If you don’t use a Fibre Channel Port Channel, you must map each interface to the correct VSAN (Figure 56).


Figure 56. Fibre Channel Port Details

If you use a Fibre Channel port channel, the VSAN mapping is part of the Fibre Channel port-channel configuration

(Figure 57).

Figure 57. Fibre Channel Port-Channel Details

If the solution uses internal Cisco Nexus 5500 platform switches to attach the Fibre Channel storage, the VSAN

configuration must match the configuration in Cisco UCS, and the zoning must be based on global best practices.

An example of a Cisco Nexus 5500 platform configuration (based on NX55XX-A) follows.

Show the defined VSAN on the Cisco Nexus switch:

NX55XX-A# show vsan

vsan 1 information

name:VSAN0001 state:active

interoperability mode:default

loadbalancing:src-id/dst-id/oxid

operational state:down

vsan 10 information

name:VSAN0010 state:active

interoperability mode:default

loadbalancing:src-id/dst-id/oxid

NX55XX-A# show vsan membership

vsan 1 interfaces:

vsan 10 interfaces:

fc2/1 fc2/2 fc2/3 fc2/4

fc2/5 fc2/6 fc2/7 fc2/8


fc2/9 fc2/10 fc2/11 fc2/12

fc2/13 fc2/14 fc2/15 fc2/16

Show the connection status for all Fibre Channel ports on the Cisco Nexus switch:

NX55XX-A# show interface brief

-------------------------------------------------------------------------------

Interface Vsan Admin Admin Status SFP Oper Oper Port

Mode Trunk Mode Speed Channel

Mode (Gbps)

-------------------------------------------------------------------------------

Fc2/1 10 auto off up swl F 8 --













Fc2/14 10 auto off notConnected swl -- --


Fc2/16 10 auto off notConnected swl -- --

Show the defined zone set and zones on the Cisco Nexus switch. In the Cisco solution for SAP HANA, a zoning

configuration based on the vHBA WWPN and the Cisco Nexus Fibre Channel ports is used.

NX55XX-A# show zoneset

zoneset name HANA-T01 vsan 10

zone name hana01-vnx1 vsan 10

pwwn 20:00:00:25:b5:01:0a:ff [hana01-vhba1]

interface fc2/1 swwn 20:00:00:05:73:e7:10:80





pwwn 20:00:00:25:b5:01:0a:df [hana02-vhba1]







pwwn 20:00:00:25:b5:01:0a:ef [hana03-vhba1]






pwwn 20:00:00:25:b5:01:0a:bf [hana04-vhba1]






pwwn 20:00:00:25:b5:01:0a:cf [hana05-vhba1]





…

<SNIP>

SAN-Based Fibre Channel Storage

SAN-based Fibre Channel storage is the most common option for connecting enterprise storage systems from

different storage vendors to Cisco UCS (Figure 58). Check the Cisco UCS Hardware Compatibility Matrix to

determine whether the storage model is supported.


Figure 58. SAN-Based Fibre Channel Storage

You must configure the fabric interconnects using the end-host Fibre Channel mode to work with Fibre Channel

switches from different vendors. The SAN zoning must be implemented on the SAN switches.

Note: If internal Cisco Nexus switches are used to connect the storage device, the Fibre Channel configuration

should work without any changes.

Note: To connect Cisco UCS to the SAN switches, you must enable N-Port ID Virtualization (NPIV) support on

all SAN devices. If NPIV is enabled and the VSAN configuration in Cisco UCS maps the configuration in the SAN,

you can connect the cables.


To verify that everything is configured correctly, select the fabric interconnect’s General tab (Figure 59).

Figure 59. Fabric Interconnect Details


Check the Fibre Channel port details to verify that the correct VSAN is selected and that the Overall Status field for

the port displays Up (Figure 60).

Figure 60. Fibre Channel Port Details

If you use a dedicated backup SAN or something similar, you need two additional vHBA templates for the backup

SAN (Figure 61).



A SAN connectivity policy is used to specify in a single place the type and order of the vHBAs that all SAP HANA

servers will use (Figure 62).

Figure 62. Creating SAN Connectivity Policy

The SAN connectivity policy configuration includes all vHBAs that are used for SAP HANA. The vHBAs are

mapped to a vHBA template, configured previously in this document (Figure 63).

Figure 63. Creating a vHBA

On the Storage tab of the service profile template, choose the defined SAN connectivity policy HANA-Order. All

vHBAs configured in the SAN connectivity policy are automatically defined in the correct order and mapped to the

correct fabric (Figure 64).


Figure 64. Service Profile Template: Storage Tab (New)

The storage must be attached to the same VSAN to establish communication between the server and the storage.

Direct-Attached Fibre Channel Storage

The direct-attached Fibre Channel storage configuration (Figure 65) is used for the Cisco Starter Kit Solution for

SAP HANA. Check the Cisco UCS Hardware Compatibility Matrix to determine whether the storage model

supports direct attachment to the fabric interconnects.


Figure 65. Direct-Attached Fibre Channel Storage

You must configure the fabric interconnects using the Fibre Channel switch mode to work with direct-attached

Fibre Channel storage from different vendors. If Cisco UCS is running in Fibre Channel end-host mode, select Set

FC Switching Mode and set FC Mode to Switch (Figure 66). Both fabric interconnects will reboot to activate the

setting. The SAN zoning in this case must be performed in Cisco UCS.


Figure 66. Fabric Interconnect Details

After the fabric interconnects have booted, you must configure the ports with the direct-attached storage as Fibre

Channel storage ports (Figure 67).

Figure 67. Configuring Fibre Channel Storage Ports


The VSAN for direct-attached storage is the default (1) VSAN (Figure 68).

Figure 68. Fibre Channel Storage Ports

Storage Connection Policy: Zoning

SAN zoning in Cisco UCS is defined by the storage-connection policies on the SAN tab. For the Zoning Type

option, select Single Initiator Multiple Targets (Figure 69).

Figure 69. Storage Connection Policy: Start


Add all WWPNs to the policy to which the SAP HANA nodes must be connected. The examples in Figures 70

through 74 use one storage resource with two controllers and allow cross-communication between each vHBA and

each storage controller.

Figure 70. Fabric A: Storage Controller 1

Figure 71. Fabric A: Storage Controller 2

Figure 72. Fabric B: Storage Controller 1


Figure 73. Fabric B: Storage Controller 2

The policy should now look like Figure 74.

Figure 74. Storage Connection Policy

You must create or change a SAN connectivity policy to use the new defined storage connection policy (Figure 75).


Figure 75. vHBA Initiator Groups List

In the vHBA Initiator Groups list, you must define an entry. Click the plus (+) button on the right side of the window.

For direct-attached storage access, only the vHBAs for data access are required; other vHBAs, such as those for

backup, are not used here. The storage connection policy defined previously must be selected (Figure 76).

Figure 76. Creating a vHBA Initiator Group


The service profile template will be updated automatically if the SAN connectivity policy was used before. If it has

not been used before, select the correct policy in the service profile template (Figure 77).

Figure 77. vHBA Initiator Group Configuration in Service Profile Template and Service Profiles

Fibre Channel–over–Ethernet Storage Options

Cisco UCS is an FCoE-based architecture that by default supports FCoE as a storage access protocol. Multiple

storage devices with FCoE support are available. This section provides an overview of how to configure Cisco UCS

to work with FCoE attached storage.

LAN-Attached FCoE Storage

Before you use the LAN-attached FCoE storage option (Figure 78), verify that all devices that are required to

establish the FCoE communication between Cisco UCS and the storage support the multihop FCoE feature.


Figure 78. LAN-Attached FCoE Storage

The configuration here uses Port Channel 20 for the normal network traffic and Port Channel 10 for the FCoE

traffic. On the Cisco UCS side, the base configuration for both Port Channels is the same. However, on the Cisco

Nexus 5500 platform side, Port Channel 20 is configured as a virtual Port Channel (vPC), and Port Channel 10 is

configured as a regular Port Channel for the Cisco Nexus 5500 platform switch.

Cisco UCS uses FCoE by default, and each VSAN is automatically mapped to an FCoE VLAN ID. This FCoE

VLAN ID must match the VLAN ID on the data center network switches. Verify, and if required change, the VLAN

ID for each VSAN that will be used (Figure 79).


Figure 79. VSAN Details: Verifying the FCoE VLAN ID

FCoE Port-Channel Configuration

You can use dedicated uplink ports or Port Channels for FCoE traffic to the next switch. Cisco UCS also supports

unified traffic on the uplink ports. Verify that you can use dedicated or unified traffic because not all switches

support unified traffic.

Note: If you use unified traffic over a Port Channel together with Cisco Nexus 5500 platform switches, vPC

configuration is not allowed. Check the Cisco Nexus 5500 platform documentation for more information.

The following steps show how to create an FCoE Port Channel in combination with a traditional Ethernet Port

Channel that will automatically define unified uplinks. You need to know the Port Channel IDs, which are on the

LAN tab (Figure 80).


Figure 80. LAN Port Channel

On the SAN tab, you must create a new FCoE Port Channel, and the ID must be the same as for the Ethernet Port

Channel (Figure 81).

Figure 81. Creating an FCoE Port Channel: Part 1

No usable ports are listed, and for this case you do not need to select any port. Click Finish (Figure 82).


Figure 82. Creating an FCoE Port Channel: Part 2


Cisco UCS automatically selects the Ethernet Port Channel and maps the FCoE Port Channel to the same

interfaces (Figure 83).

Figure 83. Creating an FCoE Port Channel: Finished


The role of the Ethernet uplink port changes from Network to Unified Uplink. Now the default Ethernet traffic and

the FCoE traffic will use the same connections (Figure 84).

Figure 84. Ethernet Port: General Tab


To build the connection between the vHBA and the FCoE Port Channel, you need to create a SAN pin group

(Figure 85).

Figure 85. SAN Pin Group

After you configure the FCoE uplink and the SAN pin group, you can change the vHBA templates to use the new

configuration (Figure 86).

Figure 86. vHBA Template Details: SAN Pin Group


Check the end-to-end configuration of all network devices for which FCoE is configured to verify that the same IDs

are used and that all links are up.

If everything is configured correctly, the server will see the configured storage resource at the next reboot.

Configuration of Internal Cisco Nexus 5500 Platform Switches

If you use the internal Cisco Nexus 5500 platform switches to attach the FCoE storage, you must change the

configuration of these switches.

The high-level changes that you need to make on the Cisco Nexus switches are shown here. Refer to the Cisco

Nexus documentation for more detailed information.

nx5k-a(config)# show run po 10

interface port-channel10

description PC to 6248-A for FCoE traffic

switchport mode trunk

switchport trunk allowed vlan 3010

spanning-tree port type edge trunk

spanning-tree bpduguard enable

nx5k-a(config)#

nx5k-a(config)# feature fcoe

nx5k-a(config)# vlan 3010

nx5k-a(config-vlan)# fcoe vsan 10

nx5k-a(config-vlan)# exit

nx5k-a(config)#

nx5k-a(config-if)# interface vfc 10

nx5k-a(config-if)# bind interface port-channel 10

nx5k-a(config-if)# no shut

nx5k-a(config-if)# exit

nx5k-a(config)# vsan database

nx5k-a(config-vsan-db)# vsan 10 interface vfc 10

nx5k-a(config-vsan-db)# exit

nx5k-a(config)#

The storage must be attached to the same VSAN or FCoE VLAN to establish communication between the server

and the storage.


Direct-Attached FCoE Storage

You can attach storage through FCoE directly to Cisco UCS (Figure 87).

Figure 87. Direct-Attached FCoE Storage

You can’t use an FCoE Port Channel for direct-attached FCoE storage. Instead, you must reconfigure the

interfaces on the fabric interconnects as FCoE storage ports (Figure 88).

Figure 88. Ethernet Port Details: Configuring a Port as an FCoE Storage Port


The port role changes from Uplink to Fcoe Storage, and the VSAN selection is available (Figure 89).

Figure 89. Ethernet Port Details: FCoE Storage

You must change the vHBA template configuration to the same VSAN as configured on the FCoE storage port to

allow communication with the direct-attached FCoE storage (Figure 90).

Figure 90. vHBA Details

If the vHBA template is an updating template, the service profile template and the service profiles automatically

update.

NFS Storage Options

As with Fibre Channel storage options, multiple NFS storage options are available.


Direct-Attached NFS Storage

With the Cisco UCS appliance port configuration, you can connect NFS storage directly to the Cisco UCS fabric

interconnects (Figure 91). Special conditions apply to this configuration if a failure occurs. The most common ones

are described in Appendix B, “Direct-Attached NFS Failure Scenarios,” in this document.

Figure 91. Direct-Attached NFS Storage


Figure 92 shows an unconfigured Appliances section on the LAN tab. No interfaces or VLANs are configured.

Figure 92. Appliances Section on the LAN Tab: No Interfaces or VLANs Configured

The first step is to configure the ports on the fabric interconnect as appliance ports. To perform this configuration,

select the Equipment tab. Select the port and choose Reconfigure > Configure as Appliance Port (Figure 93).

Figure 93. Configuring a Port as an Appliance Port


In the new window, select Trunk as the port mode and click Create VLAN (Figure 94).

Figure 94. Appliance-Port Configuration Details: No VLAN Configured


Specify a VLAN name and ID to be used for the NFS traffic (Figure 95).

Figure 95. Creating a VLAN for the Appliance Section


Select the new VLAN (Figure 96).

Figure 96. Appliance-Port Configuration Details Including the NFS Storage VLAN


After the configuration, the port role changes to Appliance Storage (Figure 97).

Figure 97. Port Details

Configure all ports to which a storage resource is attached on fabric interconnects A and B. You need to create the

VLAN only once; it is then available for selection for all other instances. The Appliances section shows all

configured ports and VLANs (Figure 98).

Figure 98. Appliance Section After Ports Are Configured

The following steps are optional, depending on your setup.


If the OS boot methodology used is PXE boot with the root file system on NFS, an additional VLAN for the boot

network is required on the appliance ports.

In the Appliances section of the LAN tab, create a new VLAN under VLANs (Figure 99).

Figure 99. Creating VLAN NFS-Boot for PXE Boot and NFS Root File System in Appliances Section


In the Appliances section, you then must add this VLAN to all ports to which boot storage is connected (Figures

100 and 101).

Figure 100. Adding the NFS-Boot VLAN to an Appliance Port


Figure 101. Appliances Section After VLAN Configuration

LAN-Attached NFS Storage

One of the most common configurations uses NFS storage with LAN switches (Figure 102). These switches can be

the data center network switches or the internal Cisco Nexus 5500 platform switches.

Figure 102. LAN-Attached NFS Storage

In the sample configuration in Figure 103, Port Channels Po20 and Po21 are used for traffic into the data center

network. All ports in the Port Channel are configured as uplink ports. Port Channels Po10 and Po11 are used for

internal traffic to the internal Cisco Nexus switches.


Figure 103. Port and Port Channel Configuration


The default configuration for the Cisco solution for SAP HANA uses the internal Cisco Nexus switches for storage

traffic. As shown in Figure 104, the internal zone includes all storage VLANs.

Figure 104. VLAN Groups


Create a VLAN group for storage traffic to route this traffic to the data center network. Specify a name and select

all VLANs used for storage traffic (Figure 105).

Figure 105. Creating a VLAN Group


On the next screen (Figure 106), dedicated uplink ports for the storage traffic are selectable. If Port hannels are

used, click Next.

Figure 106. Creating a VLAN Group: Port Selection


Select the Port Channels to route the storage traffic. In Figure 107, Port Channels Po20 and Po21 route traffic to

the data center network.

Figure 107. Creating a VLAN Group: Port-Channel Selection


The VLAN group storage is now configured and pinned to Port Channels Po20 and Po21 for all traffic in VLAN

T01-Storage and T02-Storage (Figure 108).

Figure 108. VLAN Groups List Including the Storage Group


The storage traffic now goes out to the data center network. The VLAN IDs used in Cisco UCS must match the

VLAN IDs in the data center network. To change a VLAN ID, open the required VLAN on the LAN tab and change

the VLAN ID as required (Figure 109).

Figure 109. VLAN T01-Storage Details

You also should check the MTU size on all devices. The recommended setting is 9000 for the internode Ethernet

interface.


Check the MTU size for the storage vNIC template as shown in Figure 110.

Figure 110. T01-Storage vNIC Template Details

Connection to the Internal Cisco Nexus 5500 Platform Switch

The storage can be connected to the internal Cisco Nexus switches in two ways:

● Directly connected: Storage is connected to the Cisco Nexus switch.

● Indirectly connected: Storage is connected to the data center network, which is connected to the Cisco

Nexus switch.


Figure 111 shows the directly connected storage methodology.

Figure 111. NFS Storage Directly Attached to Solution Switches

This option is similar to the Cisco solution for SAP HANA with NFS storage. Because the internal solution switches

form a critical part of this configuration and they affect overall SAP HANA performance, configuration changes are

difficult to make.

For example, Figure 112 shows the network port configuration of the Cisco Nexus switches and the NetApp FAS

storage that Cisco uses in the appliance-like solutions.


Figure 112. Cisco Nexus Ports

The first NetApp FAS device is connected to the ports Eth1/13 and Eth1/15 on both switches.

Cisco Nexus A configuration:

interface Ethernet1/13


description NetApp1_CtrlA-e1a

untagged cos 5


switchport trunk allowed vlan 201-204



channel-group 40 mode active

no shutdown


description NetApp1_CtrlB-e1a

untagged cos 5






no shutdown


description "Storage1_CtrlA_IPV6"


untagged cos 5



vpc 40


description "Storage1_CtrlB_IPV6"


untagged cos 5



vpc 41


Cisco Nexus B configuration:


description NetApp1_CtrlA-e1b

untagged cos 5






no shutdown


description NetApp1_CtrlB-e1b

untagged cos 5






no shutdown


description "Storage1_CtrlA_IPV6"


untagged cos 5



vpc 40


description "Storage1_CtrlB_IPV6"


untagged cos 5



vpc 41


NetApp FAS configuration:

hana1a> rdfile /etc/rc

…

ifgrp create lacp dvif -b ip e1a e1b

vlan create dvif 201

ifconfig dvif-201 inet6 hana1a-st prefixlen 64 mtusize 9000 partner dvif-201

ifconfig dvif-201 nfo

ifconfig dvif partner dvif

…

hana1a>

hana1b> rdfile /etc/rc

…

ifgrp create lacp dvif -b ip e1a e1b

vlan create dvif 201

ifconfig dvif-201 inet6 hana1b-st prefixlen 64 mtusize 9000 partner dvif-201

ifconfig dvif-201 nfo

ifconfig dvif partner dvif

…

hana1b>

Storage 2 is connected to Eth1/17 (PC42) and Eth1/19 (PC43).

Storage 3 is connected to Eth1/21 (PC44) and Eth1/23 (PC44).

The configuration on the Cisco Nexus switches and on the NetApp FAS device follows the same scheme as shown

for the first storage device.

Regardless of which storage device is used, it must support Link Aggregation Control Protocol (LACP), and the

VLAN ID must be configured properly end to end, from the storage over the network to the server.


Figure 113 shows the indirectly connected storage methodology.

Figure 113. NFS Storage Indirectly Attached to Solution Switches

This option is technically valid but difficult to implement, so the LAN-attached NFS storage option shown earlier in

this document is recommended.

Boot Options

The Cisco solution for the SAP HANA scale-out implementation is designed to work with PXE boot and the NFS

root file system and does not include local storage to install the operating system. Therefore, you need to define

the boot option:

● Local disk boot: You must order the required number of serial-attached SCSI (SAS) disks to store the OS

for each node.

● PXE boot with NFS root: You must provide NFS storage with redundant paths and capacity for the OS.

● SAN boot: You must provide Fibre Channel storage directly attached or SAN attached.

● iSCSI boot: You must provide LAN-attached block storage with iSCSI capability.

Each boot option has advantages and disadvantages. You should choose on option based on internal best

practices. For example, if you use SAN boot for the other servers, this option may be the best one for this

installation as well.

For PXE boot, you can use the internal management servers for Dynamic Host Configuration Protocol (DHCP) and

Trivial File Transfer Protocol (TFTP) service. All required components are installed and preconfigured for the

appliance-like solution model described in this document.

PXE Boot

For PXE boot with the Linux root file system on NFS, you need to configure the PXE server and the NFS storage

with access to the boot VLAN. Refer to the SUSE or Red Hat documentation for more information.



You must change the boot policy in the service profile template to boot from the LAN (Figure 114).

Figure 114. Boot-Policy Example for PXE Boot

SAN Boot

For SAN boot, vHBAs must be configured, and the WWPNs for the storage are required. As a best practice, the

SAN zoning should allow each HBA to see two controllers, storage processors, and front-end ports of the boot

storage. The example shown in Figure 115 uses NetApp FAS storage, and each HBA can see both NetApp FAS

controllers. Figure 116 shows a sample boot policy with EMC storage.


You must change the boot policy to enable SAN boot. If the boot logical unit number (LUN) needs to be reachable

over multiple paths, the SAN boot configuration should list the WWPNs of the storage mapped to the two vHBAs

on the server (Figures 115 and 116).


Figure 115. Boot-Policy Example for SAN Boot: NetApp FAS

Figure 116. Boot-Policy Example for SAN Boot: EMC VNX


Any change in the boot policy requires a server reboot to write the new setting in the BIOS. Without a SAN boot

device configuration in the BIOS, the VIC Fibre Channel boot driver is not loaded, and SAN boot is not possible. At

the Linux Kernel-based Virtual Machine (KVM) console, the screen shown in Figure 117 with a different WWN must

appear. If it does not appear, then check the boot policy in the service profile and the entries in the BIOS.

Figure 117. KVM Console: VIC Boot Driver Detects SAN LUN

The system is now prepared to start the OS installation. You must adopt the configuration in the OS installation

procedure, following the single-path or multipath design of the boot LUN.

Local-Disk Boot

Local-disk boot is a valid boot option if local disks are installed in all blades. Note that with local disks, the stateless

computing approach of Cisco UCS will not work. If a blade fails, the service profile cannot be easily moved from

blade A to blade B because the OS does not automatically move with the service profile.


Local-Disk Configuration Policy

The Cisco UCS B460 M4 blades come with an onboard RAID controller to manage the four disk slots. The easiest

way to configure the RAID controller is to use a local-disk configuration policy to specify the RAID type. A best

practice is to use two disks in a RAID 1 configuration to recover from a disk failure (Figure 118).

Figure 118. Local-Disk Configuration Policy



You can use the defined local-disk configuration policy in the service profile template. On the Storage tab, select

Change Local Disk Configuration Policy (Figure 119).


Select the defined policy for RAID 1. In Figure 120, it is named Mirror.

Figure 120. Service Profile Template: Changing Local Disk Configuration Policy


All service profiles mapped to this service profile template now use the new policy (Figure 121). This change

requires a server reboot to become active.

Figure 121. Service Profile Template with Local-Disk Policy

On the Boot Order tab, select a valid boot policy including a local disk (Figure 122).

Figure 122. Service Profile Template: Boot Order

To change the boot policy, click Modify Boot Policy and select or create a policy that meets your needs.


The Cisco UCS Manager default policy is preconfigured for local-disk boot. You do not need to specify a new policy

here (Figure 123).

Figure 123. Service Profile Template: Modifying Boot Policy


Cisco UCS Manager automatically reconfigures the RAID controller and creates a virtual drive as RAID 1 on the

available disks, as shown in Figure 124.

Figure 124. KVM Screen: LSI RAID Configuration Summary

You can use many other options to configure the local disks based on the RAID controller, or you can configure

them as single disks mapped to the OS with a software mirror created. The configuration shown here is the one

recommended for the local-disk boot option, but it is not the only configuration that is supported.

Operating System Installation

This document describes only the parts of the OS installation process relevant to SAP HANA. It does not provide a

detailed step-by-step guide for the whole installation process.

Note: Everyone installing an OS for SAP HANA should read all related information at http://www.saphana.com/,

the SAP HANA installation guides, and the SAP notes. This document does not necessarily reflect any changes in

the SAP requirements.

SUSE Linux Enterprise Server

This section discusses the OS installation using the local-disk boot option. The storage configuration and file-

system layout do not need to be changed because the example here uses the default SLES selection on the local

RAID 1 controller. For the SAN and PXE boot options, the storage and file-system configurations would be

different.

Note: Use the SAP HANA installation guides and follow your organization’s best practices to choose the best

file-system layout for the installation.

http://www.saphana.com/


Open the KVM console for the server to install the OS. On the Virtual Media tab, add the ISO image (Figures 125

and 126).

Figure 125. Virtual Media: Adding the Image


Figure 126. Virtual Media: Mapped SLES ISO

After the server is powered on and an ISO image is mapped to the virtual CD, the installation media is used to boot

the system. In the example in Figure 127, the default installation procedure is used.


Figure 127. Boot Selection Screen

Follow the instructions on the screen and use your internal recommended options and best practices to proceed

with the installation. Figures 128 through 133 show only the parts of the screen with specific settings recommended

by SAP or Cisco. Some settings are required by SAP and documented in the SAP HANA installation guides, and

some are only recommendations based on the solution tests.


If PXE boot is used, ignore the error message and click OK (Figure 128).

Figure 128. Error: No Hard Disk Found


SAP recommends setting the server time zone on all SAP HANA nodes to UTC (Figure 129), for universal time.

Every user configured on the system can have an individual time-zone setting to use the local time.

Figure 129. Clock and Time Zone


As a best practice, change the default run level from 5 to 3 for all types of servers (Figure 130).

Figure 130. Setting the Default Run Level


You must select SAP HANA Server Base on the Software Selection and System Tasks screen, and Cisco

recommends that you deselect the Gnome Desktop Environment package on the same screen (Figure 131).

Figure 131. Software Selection


Figure 132 shows a summary page for this installation with the required software packages and the default run

level set to 3 instead of 5.

Figure 132. Installation Settings Summary: Software Packages


With the local-disk boot option, the partitioning can be simple. Use internal best practices to define the partitioning

that best fits your use case (Figure 133).

Figure 133. Installation Settings Summary: Partitioning for Local-Disk Boot


For the PXE boot option, you must configure the partitioning manually. One NFS share is used with a mount point

(Figure 134).

Figure 134. NFS Root File System

Skip the network configuration for now; this configuration is discussed later in this document.


For a PXE boot, you must configure the network interface card (NIC) for the initial boot with DHCP (Figure 135). To

identify the correct interface, refer to the section “Network Configuration Options” later in this document.

Figure 135. Network Configuration


The initial user authentication method should be Local (/etc/passwd). You can add methods later if required (Figure

136).

Figure 136. User Authentication Method


The system will boot the OS. You may encounter some failure messages, such as the network failure message

shown in Figure 137, if all the interfaces are not configured yet.

Figure 137. Linux Login Screen

Operating System Configuration

To run SAP HANA on a SLES 11 SP2 or SP3 system, you need to make some configuration changes at the OS

level to provide the best performance and a stable system.

Configure OS Settings for Console Redirection

Add or uncomment the following in /etc/inittab:

se:2345:respawn:/sbin/agetty 115200 ttyS0

Add the following value to /etc/securetty:

ttyS0

Configure the file /tftpboot/pxelinux.cfg/<IP in HEX>.

Append the following text to the APPEND line:

console=tty1 console=ttyS0,115200


Here is an example of a modified /tftpboot/pxelinux.cdf/C0A87F5B file in SLES 11 SP2:

mgmtsrv01:/tftpboot/pxelinux.cfg # gethostip server01

server01 192.168.127.91 C0A87F5B

mgmtsrv01:/tftpboot/pxelinux.cfg # ls –l

lrwxrwxrwx 1 root root 8 Jan 25 09:11 192.168.127.91 -> C0A87F5B

-rw-r--r-- 1 root root 680 Feb 25 22:41 C0A87F5B

lrwxrwxrwx 1 root root 8 Jan 25 09:11 server01 -> C0A87F5B

mgmtsrv01:/tftpboot/pxelinux.cfg # cat C0A87F5B

# SAP UCS PXE Boot Definition

display ../boot.msg

default SLES11_SP2

prompt 1

timeout 10

LABEL SLES11_SP2

KERNEL vmlinuz-default

APPEND initrd=initrd_cisco.gz rw rootdev=192.168.127.11:/FS_OS_01/SLES11SP2

rootfsopts=default intel_idle.max_cstate=0 ip=dhcp console=tty1

console=ttyS0,115200 crashkernel=256M-:4G

With these settings, console redirection for SLES is configured (Figures 138 to 141).

Figure 138. Logging in to Serial Console


Figure 139. Serial Console Power-On Self-Test (POST) Screen

Figure 140. Serial Console Boot Menu


Figure 141. Serial Console OS Booted

Enable Linux Kernel Crash Dump

In the test, the Magic SysRq feature was enabled permanently by editing /etc/sysconfig/sysctl and changing the

ENABLE_SYSRQ line to ENABLE_SYSRQ="yes". This change becomes active after a reboot.

# vi /etc/sysconfig/sysctl

…

#

# Magic SysRq Keys enable some control over the system even if it

# crashes (e.g. during kernel debugging).

#

# Possible values:

# - no: disable sysrq completely

# - yes: enable all functions of sysrq

# - bitmask of allowed sysrq functions:

# 2 - enable control of console logging level

# 4 - enable control of keyboard (SAK, unraw)

# 8 - enable debugging dumps of processes etc.

# 16 - enable sync command

# 32 - enable remount read-only

# 64 - enable signalling of processes (term, kill, oom-kill)

# 128 - allow reboot/poweroff

# 256 - allow nicing of all RT tasks

#

# For further information see /usr/src/linux/Documentation/sysrq.txt

#

ENABLE_SYSRQ="yes"

To enable the feature for the running kernel, enter this command:


# echo 1>/proc/sys/kernel/sysrq

Capture Kernel Core Dumps

The test followed the SLES guidelines and adopted the changes described here to capture kernel core dumps:

● Install the packages kdump, kexec-tools, and makedumpfile.

● Reserve memory for the captured kernel in the boot menu /pxelinux.cfg.

Here is an example of a modified /tftpboot/pxelinux.cfg/C0A87F5B file in SLES 11 SP2:

mgmtsrv01:/tftpboot/pxelinux.cfg # gethostip server01

server01 192.168.127.91 C0A87F5B

mgmtsrv01:/tftpboot/pxelinux.cfg # ls –l

lrwxrwxrwx 1 root root 8 Jan 25 09:11 192.168.127.91 -> C0A87F5B

-rw-r--r-- 1 root root 680 Feb 25 22:41 C0A87F5B

lrwxrwxrwx 1 root root 8 Jan 25 09:11 server01 -> C0A87F5B

mgmtsrv01:/tftpboot/pxelinux.cfg # cat C0A87F5B

# SAP UCS PXE Boot Definition

display ../boot.msg

default SLES11_SP2

prompt 1

timeout 10

LABEL SLES11_SP2

KERNEL vmlinuz-default

APPEND initrd=initrd_cisco.gz rw rootdev=192.168.127.11:/FS_OS_01/SLES11SP2

rootfsopts=default intel_idle.max_cstate=0 ip=dhcp console=tty1

console=ttyS0,115200 crashkernel=256M-:4G

Activate the kdump system service:

Run

mgmtsrv01:# chkconfig boot.kdump on

Instead of a local dump destination, an NFS share with enough space to store the crash dump files was used.


Add the network device to be used to the variable KDUMP_NETCONFIG in /etc/sysconfig/kdump:

## Type: string

## Default: "file:///var/log/dump"

## ServiceRestart: kdump

#

# Which directory should the dumps be saved in by the default dumper?

# This can be:

#

# - a local file, example "file:///var/log/dump" (or, deprecated,

# just "/var/log/dump")

# - a FTP server, for example "ftp://user:passwd@host/var/dump"

# - a SSH server, for example "ssh://user:passwd@host/var/dump"

# - a NFS share, for example "nfs://server/export/var/log/dump"

# - a CIFS (SMB) share, for example

# "cifs://user:passwd@host/share/var/dump"

#

# See also: kdump(5) which contains an exact specification for the URL format.

# Consider using the "yast2 kdump" module if you are unsure.

#

KDUMP_SAVEDIR="nfs://192.168.127.11/FS_crash"

Note: Frequently check to be sure that KDUMP_SAVEDIR has enough space to prevent the system from

hanging while it waits for the kdump procedure to complete. As described at

http://www.novell.com/support/kb/doc.php?id=3374462, you can trigger a kernel core dump manually through a

Magic SysRq keyboard combination. This dump can be helpful if the system is hanging.

Configure Magic SysRq Macros in Cisco UCS Manager

In Cisco UCS Manager, right-click and choose Service Profile > KVM Console > User Defined Macros. Create the

Magic SysRq keyboard combination as shown in Figure 142.

Figure 142. Configuring User-Defined Macros

In the test, Magic SysRq macros were created for Emergency Sync and for Kernel Crash Dump. You can use

Emergency Sync to check whether the SysRq function is enabled and configured.

http://www.novell.com/support/kb/doc.php?id=3374462


After Emergency Sync is initiated, the message shown in Figure 143 should appear in the console.

Figure 143. SysRq Emergency Sync Console Message

Configure Network Options

To configure the network, you need to know what Ethernet interface in the OS is mapped to which vNIC in Cisco

UCS. This mapping is easy to identify using the MAC addresses.

Configure Cisco UCS B440 M2 with One VIC Installed

Figures 144 and 145 show the configuration for the Cisco UCS B440 M2 with only one VIC installed.

Figure 144. Equipment > Servers with General Tab and Adapter Folder Open


Figure 145. Servers > Service Profile XYZ with Network Tab Open

Compare the MAC addresses of the Cisco UCS Manager vNIC configuration with the Ethernet interfaces in the

OS. Because only one VIC is used, the order is 1 to 9 and matches exactly the order for the OS (Figure 146).

Figure 146. Linux: ifconfig –a |grep HWaddr


Configure Cisco UCS B440 M2 with Two VICs Installed

Figure 147 shows the configuration for the Cisco UCS B440 M2 with two VICs installed.

Figure 147. Equipment > Servers with General Tab Open

In Figure 148, the Actual Placement column shows the VIC to which the vNIC is mapped, and the Actual Order

column shows the order for each VIC. Note that SLES uses the order when scanning the PCI subsystem and will

select the second VIC before the first VIC.


Figure 148. Servers > Service Profile XYZ with Network Tab Open


As Figure 149 shows, eth0 has the MAC address 00:25:B5:14:00:AF, which is used by the backup vNIC placed on

VIC 2 followed by the client vNIC with MAC address 00:25:B5:13:00:5F.

Figure 149. Linux: ifconfig –a |grep HWaddr

This behavior is important to know if the OS installation or configuration is automated using a deployment or

workflow tool.

Configure Cisco UCS B460 M4 with Two VICs Installed

The Cisco UCS B460 M4 has four mezzanine slots and uses four adapters: two Cisco UCS VIC 1240 adapters

with a port-expander card, and two Cisco UCS VIC 1280 adapters. The eight vNICs (one for each network with

failover enabled, as described earlier) and two vHBAs (one per fabric) are distributed across vCONs as shown in

Figure 150.


Figure 150. Cisco UCS B460 M4 with Four VICs: Example

Compare the MAC addresses of the Cisco UCS Manager vNIC configuration with the Ethernet interfaces in the OS

as shown in Figure 151. This information is useful for configuring the correct IP addresses for the Ethernet

interfaces according to the VLAN configuration and IP address information captured earlier.

In the example shown here, the backup vNIC in Cisco UCS Manager is eth1 at the OS level. As shown in the

sample VLAN and IP address list, the IP address can be assigned based on the node number, starting with, for

example, 172.25.221.102 for this first node. The IP addresses for the rest of the Ethernet interfaces are assigned.

The same procedure is used for configuring the Ethernet IP addresses on the rest of the SAP HANA nodes as they

are installed.


Figure 151. vNIC, MAC Address, and Ethernet Interface Mapping

Configure the Default Router

Add the required entry in /etc/sysconfig/network/routes:

default <ROUTER IP> - -

Disable Transparent Huge Pages

With SLES 11 SP2, the use of transparent huge pages (THP) is generally activated for the Linux kernel. THP

allows multiple pages to be handled as huge pages, reducing the translation-lookaside-buffer (TLB) footprint in

situations in which THP may be useful. Because of the way that SAP HANA manages memory, the use of THP

may lead to system hang and performance degradation.

To disable the use of THP, specify the kernel settings at runtime as follows:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

You do not need to shut down the database to apply this configuration. This setting is then valid until the next

system start. To make this option persistent, integrate this command line into your system boot scripts (for

example, /etc/init.d/after.local).

Configure C-States for Lower Latency in Linux

Linux Kernel 3.0 includes a new cpuidle driver for recent Intel CPUs: intel_idle. This driver leads to a different

behavior in C-state switching. The normal operating state is C0, and when the processor is placed in a higher C

state, it will save power. However, for low-latency applications, the additional time needed to begin processing the

code again will cause performance degradation.

Therefore, you should edit the boot-loader configuration. The location of the boot-loader configuration file is usually

/etc/sysconfig/bootloader. Edit this file and append the following value to the DEFAULT_APPEND parameter

value:

intel_idle.max_cstate=0


This command implements a persistent change for potential kernel upgrades and boot-loader upgrades. For an

immediate configuration change, you also need to append this parameter in the kernel command line of your

current active boot-loader file, which is located on the PXE server under /tftpboot/pxelinux.cfg.

Append the intel_idle value mentioned earlier only to the operational kernel parameter line.

The C-states are disabled in the BIOS, but to be sure that the C-states are not used, set the following parameter in

addition to the previous one:

processor.max_cstate=0

The CPU speed must be set to performance for SAP HANA so that all cores run all the time with the highest

frequency:

/usr/bin/cpupower frequency-set –g performance 2>&1

To make this option persistent, integrate this command line into your system boot scripts (for example,

/etc/init.d/after.local).

Configure Swappiness

Set swappiness to 30 to avoid swapping:

Echo 30 > /proc/sys/vm/swappiness

Configure Receive-Side Scaling and Receive-Packet Scaling in the OS

To use the RSS setting in the adapter policy, you must configure receive-packet scaling (RPS) at the OS level.

RPS distributes the load of received packet processing across multiple CPUs. Note that protocol processing

performed in the New API (NAPI) context for received packets is serialized per device queue and becomes a

bottleneck under high packet load. This characteristic substantially limits the packets-per-second (pps) rate that

can be achieved on a single-queue NIC and provides no scaling for multiple cores.

In the testing reported in this document, the best performance results were achieved with the following setting:

echo 3ff > /sys/class/net/${ethernet_device}/queues/rx-0/rps_cpus

Replace ${Ethernet_device} with all the Ethernet devices for which you need high throughput. Here are some

examples:

echo 3ff > /sys/class/net/eth0/queues/rx-0/rps_cpus




To make this option persistent, integrate this command line into your system boot scripts (for example,

/etc/init.d/after.local).

Configure Host Name

The operating system must be configured so that the short name of the server appears if the hostname command

is used. The fully qualified host name is displayed when the hostname–f command is used.

Configure Network Time

The time on all components used for SAP HANA should be the same. The configuration of the Network Time

Protocol (NTP) is important and should be performed on all systems.

cishanar01:~ # cat /etc/ntp.conf


server <NTP-SERVER IP>

fudge <NTP-SERVER IP> stratum 10

keys /etc/ntp.keys

trustedkey 1

Configure Domain Name Service

The Domain Name Service (DNS) configuration must be performed based on local requirements. Here is a sample

configuration:

cishana07:/etc/sysconfig # vi /etc/resolv.conf

search cisco-hana.corp

nameserver 10.17.121.30

nameserver 10.17.122.10

Configure Secure Shell Keys

Secure Shell (SSH) keys must be exchanged between all nodes in a SAP HANA scale-out system for the user root

and the user <SID>adm. Here is a sample configuration:

cishanar01:~/.ssh # ssh-keygen –b 2048

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

fb:66:0f:8d:fc:40:f7:c1:e6:15:46:6e:ec:5f:7d:af hadmin@mgmtsrv01


The key's randomart image is:

+--[ RSA 2048]----+

| . |

| + |

| * |

| .+ o |

| S . . +.= |

| + + + o= |

| . = . o o |

| .o+ . |

| o..o E |

+-----------------+

cishanar01:~/.ssh #

cishanar01:~/.ssh # cat id_rsa.pub

ssh-rsa

AAAAB3NzaC1yc2EAAAABIwAAAQEAuqcSAZk01nGWYpwsqgAfb4j1p0zOx4axwdFDlwa2rqRTvZ

yMIqW8ajtkiQaUInSTknUhQuaBlN90GPz9u5bkgJG8xJ7U/1l8xwd/q6NbCocJRyNIYq5JvohFmoOFb/Q

rEWhwugdGg/lEefFRPHltJm

v/wqfRgaUovf/t3Tn99gBkQIYdBEe5FoW7xx+4tt4SINjj/I8VXVS7fVRLshR7cjHHLekEzAY+g6p+tQh

yfZQ1yR1dS12wDs4UeAjcD1

6JaiJUeAb35dg/5ai3I+tLyBtOcoXuJvm0kmv7mVb925FbG1mOCXqzI1HeonQTsPbxnXw6tup7Lq+oZaK

EoZQIpQ== root@mgmtsrv01

cishanar01:~/.ssh #

cishanar01:~ # ssh-copy-id -i /root/.ssh/id_rsa.pub <OTHER HANA NODES>

Password: ********

…

Configure Sudoers

The Fibre Channel client (fcClient) requires configuration changes to run properly. These changes are performed

in the SAP HANA installation procedure.

Configure Syslog

For centralized monitoring of all SAP HANA nodes, configure syslog-ng to forward all messages to a central

syslog server.


Change the syslog-ng.conf file as follows:

cishanar02:~ # vi /etc/syslog-ng/syslog-ng.conf

…

…

…

#

# Enable this and adopt IP to send log messages to a log server.

#

destination logserver1 { udp("<SYSLOG-SERVER IP>" port(<SYSLOG-Server PORT>)); };

log { source(src); destination(logserver1); };

destination logserver2 { udp("<SYSLOG-SERVER IP>" port(<SYSLOG-Server PORT>)); };

log { source(src); destination(logserver2); };

…

…

Now restart the syslog daemon:

cishanar02:~ # /etc/init.d/syslog restart

Configure sysctl.conf

The following parameters must be set in /etc/sysctl.conf. The SAP HANA installer adds some of them, so you

should set them after SAP HANA is installed.

Following is an example:

fs.inotify.max_user_watches = 65536

kernel.shmmax = 9223372036854775807

kernel.sem = 1250 256000 100 8192

kernel.shmall = 1152921504606846720

kernel.shmmni = 524288

# SAP HANA Database

vm.max_map_count = 152500000

fs.file-max = 20000000

fs.aio-max-nr = 196608

vm.memory_failure_early_kill = 1

#

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.core.rmem_default = 262144

net.core.wmem_default = 262144

#

net.core.optmem_max = 16777216

net.core.netdev_max_backlog = 300000

net.ipv4.tcp_slow_start_after_idle = 0

net.ipv4.conf.default.promote_secondaries = 1

net.ipv4.conf.all.promote_secondaries = 1


net.ipv4.icmp_echo_ignore_broadcasts = 1

net.ipv4.conf.default.accept_source_route = 0

net.ipv4.tcp_rmem = 65536 16777216 16777216

net.ipv4.tcp_wmem = 65536 16777216 16777216

#


net.ipv4.tcp_syncookies = 0

net.ipv4.tcp_dsack = 0

#

net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_moderate_rcvbuf = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_timestamps = 1

net.ipv4.tcp_sack = 1

sunrpc.tcp_slot_table_entries = 128

# 1557506 - Memory Page Cache Linit Feature 16384 GB: 335544 (=2% of 16384 *

1024 MB)

vm.pagecache_limit_mb = 335544

vm.pagecache_limit_ignore_dirty = 1

# 2205917 - SAP HANA DB: Recommended OS settings for SLES 12

kernel.numa_balancing = 0

# Linux SAP swappiness recommendation

vm.swappiness = 10

Red Hat Enterprise Linux Installation

This section discusses the Red Hat Enterprise Linux installation using the local-disk boot option. The storage

configuration and file-system layout do not need to be changed because the example here uses the default RHEL

selection on the local RAID 1 controller. For the SAN and PXE boot options, the storage and file-system

configurations would be different.

Note: Use the SAP HANA installation guides and follow your organization’s best practices to choose the best

file-system layout for the installation.

Note: Open the KVM console for the server to install the OS, and on the Virtual Media tab add the ISO image

(Figure 152).


Figure 152. KVM Console: Virtual Media Tab

After the server is powered on and an ISO image is mapped to the virtual CD, the installation media is used to boot

the system. The example in Figure 153 uses the default installation procedure.


Figure 153. Installation Menu from ISO Boot

Follow the instructions on the screen and use your internal recommended options and best practices to proceed

with the installation. Figures 154 through 157 show only the parts of the screen with specific settings recommended

by SAP or Cisco. Some settings are required by SAP and documented in the SAP HANA installation guides, and

some are only recommendations based on the solution tests.

Note: When performing the OS installation on a SAN boot device, be sure to have the appropriate FCoE NIC

(fnic) driver ISO image as specified in the Cisco UCS Hardware and Software Interoperability Matrix for the Cisco

UCS Manager, server, OS type, and VIC combination in your setup downloaded and available. You can use the

fnic driver ISO image during installation to verify that the SAN boot devices are recognized and that there are no

device communication errors due to any driver version mismatch during or after installation. Use Install or Upgrade

an existing system with the linux mpath dd command-line option to be able to read the multipath device and to use

the correct fnic driver ISO image as the driver disk during the installation process. Refer to

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vic_drivers/install/Linux/2-

0/b_Cisco_VIC_Drivers_for_Linux_Installation_Guide/b_Cisco_VIC_Drivers_for_Linux_Installation_Guide_chapter

_010.html (see the section “Installing Linux to SAN Storage Using the fNIC Driver and OS Driver Disk”) for more

information.

http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vic_drivers/install/Linux/2-0/b_Cisco_VIC_Drivers_for_Linux_Installation_Guide/b_Cisco_VIC_Drivers_for_Linux_Installation_Guide_chapter_010.html




On the KVM tab, select Basic Storage Devices and click Next (Figure 154).

Figure 154. Selecting the Storage Device Type


SAP recommends setting the server time zone on all SAP HANA nodes to UTC (Figure 155). Every user

configured on the system can have an individual time-zone setting to use the local time.

Figure 155. Selecting a Time Zone


With the local-disk boot option, the partitioning can be simple. Use internal best practices to define the partitioning

that best fits your use case. The Create Custom Layout option is shown in Figure 156.

Figure 156. Choosing a Disk Layout

Table 10 shows an example of disk partitioning using the logical volume manager (LVM).

Table 10. Sample Partitioning with LVM

File System Name Mount Point Type

/dev/sda1 /boot ext3

/dev/rootvg/rootvol / ext3

/dev/rootvg/swapvol swap swap

/dev/rootvg/sapmnt /hana/shared ext3

/dev/rootvg/sapdatavol /hana/data xfs

/dev/rootvg/backupvol /backup ext3

/dev/rootvg/recovervol /recover ext3

/dev/rootvg/saplogvol /hana/log xfs


The installation will create the partition, volume groups, and logical volumes and then await your input. The installer

starts the installation process; wait until it is complete. When the installation is complete, reboot the server. Click

Reboot and unmount the ISO image from the KVM console (Figure 157).

Figure 157. Installation Complete

Operating System Configuration

To run SAP HANA on an RHEL system, you need to make some configuration changes at the OS level to provide

the best performance and a stable system.

Edit NTP Server Configuration

Edit the file /etc/ntp.conf to reflect the appropriate NTP servers for the region and start the NTP service:

[root@cishanar04-m4 ~]# service ntpd stop

[root@cishanar04-m4 ~]# ntpdate ntp.example.com

[root@cishanar04-m4 ~]# service ntpd start

[root@cishanar04-m4 ~]# chkconfig ntpd on

[root@cishanar04-m4 ~]# chkconfig ntpdate on

[root@cishanar04-m4 ~]# chkconfig | grep ntpd

ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

ntpdate 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Disable Security-Enhanced Linux

Security-Enhanced Linux (SELinux) is an implementation of a mandatory access control mechanism in the Linux

kernel. It checks for allowed operations after standard discretionary access controls are checked. It was created by


the U.S. National Security Agency (NSA) and can enforce rules on files and processes and their actions in a Linux

system, based on defined policies.

For SAP HANA, you must disable SELinux:

[root@[root@cishanar04-m4 ~]# cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing - SELinux security policy is enforced.

# permissive - SELinux prints warnings instead of enforcing.

# disabled - No SELinux policy is loaded.

SELINUX=disabled

# SELINUXTYPE= can take one of these two values:

# targeted - Targeted processes are protected,

# mls - Multi Level Security protection.

SELINUXTYPE=targeted

Install Additional Packages

The system also requires additional packages to support SAP HANA. Install the additional packages as follows:

[root@cishanar04-m4 ~]# yum install gtk2 libicu xulrunner ntp sudo tcsh libssh2

expect cairo graphviz iptraf krb5-workstation krb5-libs.i686 nfs-utils lm_sensors

rsyslog openssl098e openssl Packagekit-gtk-module libcanberra-gtk2 libtool-ltdl

xauth compat-libstdc++-33 numactl

Disable Kernel Dump Service

Disable the kernel dump service as follows:

[root@cishanar04-m4~]# service kdump stop

[root@cishanar04-m4~]# chkconfig kdump off

Install compat-sap-c++

Install the most important package, compat-sap-c++, from RHEL for SAP HANA:

[root@cishanar04-m4~]# rpm –ivh compat-sap-c++-4.7.2-10.el6_5.x86_64.rpm

Tune the SAP HANA Profile

Install and configure the package tuned-profiles-sap-hana from RHEL for SAP HANA:

[root@cishanar04-m4~]# rpm –ivh tuned-profiles-sap-hana-0.2.19-

13.el6_6.1.noarch.rpm

[root@cishanar04-m4~]# tuned-adm profile sap-hana

Configure sysctl.conf

The following parameters must be set in /etc/sysctl.conf. The SAP HANA installer adds some of them, so you

should set them after SAP HANA is installed.

Following is an example:

fs.inotify.max_user_watches = 65536

kernel.shmmax = 9223372036854775807

kernel.sem = 1250 256000 100 8192


kernel.shmall = 1152921504606846720

kernel.shmmni = 524288

# SAP HANA Database

vm.max_map_count = 152500000

fs.file-max = 20000000

fs.aio-max-nr = 196608

vm.memory_failure_early_kill = 1

#

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.core.rmem_default = 262144

net.core.wmem_default = 262144

#

net.core.optmem_max = 16777216

net.core.netdev_max_backlog = 300000

net.ipv4.tcp_slow_start_after_idle = 0

net.ipv4.conf.default.promote_secondaries = 1

net.ipv4.conf.all.promote_secondaries = 1

net.ipv4.icmp_echo_ignore_broadcasts = 1


net.ipv4.tcp_rmem = 65536 16777216 16777216

net.ipv4.tcp_wmem = 65536 16777216 16777216

#


net.ipv4.tcp_syncookies = 0

net.ipv4.tcp_dsack = 0

#

net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_moderate_rcvbuf = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_timestamps = 1

net.ipv4.tcp_sack = 1

sunrpc.tcp_slot_table_entries = 128

# 1557506 - Memory Page Cache Linit Feature 16384 GB: 335544 (=2% of 16384 *

1024 MB)

vm.pagecache_limit_mb = 335544

vm.pagecache_limit_ignore_dirty = 1

# 2205917 - SAP HANA DB: Recommended OS settings for SLES 12

kernel.numa_balancing = 0

# Linux SAP swappiness recommendation

vm.swappiness = 10

If the settings are changed during normal operation, they can be activated by calling sysctl –p.

For compatibility reasons, four symbolic links are required:

# ln -s /usr/lib64/libssl.so.0.9.8e /usr/lib64/libssl.so.0.9.8


# ln -s /usr/lib64/libssl.so.1.0.1e /usr/lib64/libssl.so.1.0.1

# ln -s /usr/lib64/libcrypto.so.0.9.8e /usr/lib64/libcrypto.so.0.9.8

# ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.1.0.1

Configure Transparent Huge Pages

Because SAP HANA can crash occasionally when transparent huge pages are enabled,

they should be deactivated until you have verified the system. In the file

/boot/grub/grub.conf, add the following kernel command-line argument:

transparent_hugepage=never

Configure Crash Dump

All crashes of SAP HANA are handled by SAP support directly. They don't rely on OS mechanisms at all.

Therefore, you can omit the application crash and core file handling of the OS.

The abrt service that handles application crashes needs to be disabled completely:

# chkconfig abrtd off

# chkconfig abrt-ccpp off

# service abrtd stop

# service abrt-ccpp stop

The core file creation also needs to be disabled. To disable core dumps for all users, open /etc/security/limits.conf

and add the following:

* soft core 0

* hard core 0

Adaptation of the Basic SAP Notes Settings

Edit /boot/grub/menu.lst and append the following parameter to the kernel line as specified in the SAP notes:

intel_idle.max_cstate=0

Configure OS Settings for Console Redirection

Add the following value to /etc/securetty:

ttyS1

Configure the file /boot/grub/grub.conf.

Append the following text to the KERNEL line:

console=tty0 console=ttyS1,115200n8

From vi, open /etc/init/ttyS1.conf and append the following configuration options:

start on runlevel [345]

stop on runlevel [S016]

respawn

instance /dev/ttyS1

exec /sbin/agetty ttyS1 115200 vt100-nav

Reboot the OS.


Storage Access for SAP HANA

This section presents basic information about the configuration of SAP HANA storage at the OS level. The

underlying infrastructure configuration was discussed earlier in this document.

The information presented here describes examples based on the Cisco solution for SAP HANA and provides only

a high-level overview of the configuration process. You must work with your storage vendor to define the best

configuration based on your storage model and use case.

Block Storage for SAP HANA Data and Log Files

The block-storage configuration in the OS for data and log files is the same for all Fibre Channel solutions

regardless of whether the traffic traverses native Fibre Channel or FCoE.

Linux Multipathing

For block storage, you should use a multipath configuration. This section provides an example of this configuration

using the Cisco and EMC solution for SAP HANA and native Linux multipathing (device manager multipath I/O

[DM-MPIO]) on the SAP HANA nodes to improve performance and provide high availability for the access paths to

the storage devices.

Figure 158 shows the multipath relationship between a LUN in the storage array and its corresponding single- and

multipath devices on the SAP HANA node.

Figure 158. DM-MPIO with Persistent Group Reservation (PGR) on EMC VNX5300

A LUN in the EMC VNX array belongs to one storage processor (the default storage processor), but it can move

(trespass) to the other storage processor if the default storage processor fails. Each storage processor has four

Fibre Channel I/O ports (ports 4 to 7). On the other side, each SAP HANA node has two vHBAs (vHBA1 and

vHBA2), with each vHBA having two paths to each storage processor. Therefore, there are four active paths from

the SAP HANA node to each LUN, and four enabled paths. The enabled paths will become active if a trespass of

the LUN in the storage array occurs.


On the host, the LUN is visible over eight paths—four active paths and four enabled paths—and is represented by

a single device for each path: for example, /dev/sdar. The single devices are combined into a multipath device: dm-

2.

The following multipath command illustrates the example in Figure 158:

$ multipath –ll

3600601603fa02900a0e80ccf671ee111 dm-6 DGC,VRAID

size=2.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=rw

|-+- policy='round-robin 0' prio=4 status=active

| |- 2:0:4:3 sdx 65:112 active ready running

| |- 2:0:5:3 sdag 66:0 active ready running

| |- 1:0:2:3 sdar 66:176 active ready running

| `- 1:0:3:3 sdba 67:64 active ready running

`-+- policy='round-robin 0' prio=1 status=enabled

|- 2:0:2:3 sdf 8:80 active ready running

|- 2:0:3:3 sdo 8:224 active ready running

|- 1:0:4:3 sdbj 67:208 active ready running

`- 1:0:5:3 sdbs 68:96 active ready running

The specific configuration requirements depend on the storage vendor and model and must be provided by the

storage vendor.

DM-MPIO requires a configuration file /etc/multipath.conf. A multipath.conf file sample for a VNX array with the

active-passive PNR failover mode 1 method for storage host registration is shown here:

cishana01:~ # cat /etc/multipath.conf

defaults {

user_friendly_names no

}

devices {

device {

vendor "DGC"

product ".*"

product_blacklist "LUNZ"

features "0"

hardware_handler "1 emc"

path_selector "round-robin 0"

path_grouping_policy group_by_prio

failback immediate

rr_weight uniform

no_path_retry 5

rr_min_io_rq 1

path_checker emc_clariion

prio emc

}

}


A sample multipath.conf file for VNX array and host registration in Asymmetric Logical Unit Access (ALUA) mode is

shown here:

defaults {

checker_timeout 60

user_friendly_names no

}

devices {

device {

vendor "DGC"

product ".*"

product_blacklist "LUNZ"

path_grouping_policy group_by_prio

prio emc

hardware_handler "1 alua"

features "1 queue_if_no_path"

no_path_retry 60

path_checker emc_clariion

failback immediate

flush_on_last_del yes

fast_io_fail_tmo off

dev_loss_tmo 120

}

}

Consult with the storage vendor for case-specific multipath.conf files.

Fdisk

In most cases, one partition per LUN and no volume management is used. The exact partitioning and configuration

must be defined by the storage vendor based on the storage model and the use case.

File-System Type

To specify the file system used for the SAP HANA data and log volumes, consult the storage vendor to determine

the best performance and availability.

The recommended approach is to use the storage connector API for the SAP HANA data and log volumes and not

/etc/fstab. Check with the storage vendor to learn how the volumes operate: with fstab or global.ini.

Global.ini

The SAP HANA storage connector API for block storage is responsible for remounting and performing I/O fencing

for the SAP HANA persistent layer. It must be used in a SAP HANA scale-out installation in which the persistence

resides on block-attached storage devices.

Implement the API by enabling the appropriate entry in the SAP HANA global.ini file. This file resides in the

/hanamnt/shared/>SID>/global/hdb/custom/config directory.

An example of a global.ini file for EMC VNX storage is shown here:

[persistence]


use_mountpoints = yes

basepath_datavolumes = /hana/data/ANA

basepath_logvolumes = /hana/log/ANA

basepath_shared=yes

[storage]

ha_provider = hdb_ha.fcClient

partition_*_*__prType = 5

partition_1_data__wwid = 3600601603fa02900bad499d33b02e211

partition_1_log__wwid = 3600601603fa0290072085bfb3e02e211

partition_2_data__wwid = 3600601603fa0290044191eb83b02e211

partition_2_log__wwid = 3600601603fa02900ba553c933f02e211

partition_3_data__wwid = 3600601603fa02900cc468dee3f02e211

partition_3_log__wwid = 3600601603fa02900b8a136174002e211

partition_4_data__wwid = 3600601603fa02900ec2412d53f02e211

partition_4_log__wwid = 3600601603fa0290070aa76024002e211

partition_5_data__wwid = 3600601603fa02900882482504002e211

partition_5_log__wwid = 3600601603fa02900be525cf84002e211

partition_6_data__wwid = 3600601603fa029000c97cc324002e211

partition_6_log__wwid = 3600601603fa029006208f9de4002e211

For the exact configuration of the [storage] section in the global.ini file, you must consult the storage vendor.

File Storage for SAP HANA Data and Log Files

If you use NFS for the SAP HANA data and log files, you must configure the solution properly to provide good

performance and high availability for SAP HANA.

High-Availability Configuration

Starting from SAP HANA Revision 35, the ha_provider Python class supports the STONITH function. With this

Python class, you can reset the failing node to prevent a “split brain” and thus an inconsistency in the database.

Even with NFSv4, there is some minimal, theoretical risk that such a situation may occur. A reset of the failed node

eliminates this theoretical risk.

Here is an example of a configuration for ucs_ha_class.py and ucs_ipmi_reset.sh:

cishanar01:/ # cd /hana/shared

cishanar01:/hana/shared # mkdir scripts

cishanar01:/hana/shared # cd scripts

cishanar01:/hana/shared/scripts # vi ucs_ha_class.py

"""

Function Class to call the reset program to kill the failed host and remove NFS

locks for the SAP HANA HA

Class Name ucs_ha_class

Class Phath /usr/sap/<SID>/HDB<ID>/exe/python_support/hdb_ha

"""

from client import StorageConnectorClient

import os


class ucs_ha_class(StorageConnectorClient):

def __init__(self, *args, **kwargs):

super(ucs_ha_class, self).__init__(*args, **kwargs)

def stonith(self, hostname):

os.system ("/bin/logger STONITH HANA Node:" + hostname)

os.system ("/hana/shared/scripts/ucs_ipmi_reset.sh " + hostname)

def about(self):

ver={"provider_company":"Cisco",

"provider_name" :"ucs_ha_class",

"provider_version":"0.1",

"api_version" :1}

self.tracer.debug('about: %s'+str(ver))

print '>> ha about',ver

return ver

def sudoers():

return """ALL:NOPASSWD: /bin/mount, /bin/umount, /bin/logger"""

def attach(self,storages):

pass

def detach(self, storages):

pass

def info(self, paths):

pass

cishanar01:/hana/shared/scripts #


cishanar01:/hana/shared/scripts # vi ucs_ipmi_reset.sh

#!/bin/bash

# SAP HANA High Availability

# Version 1.0 08/2012

if [ -z $1 ]

then

exit 1

fi

/bin/logger `whoami`" Resetting the HANA Node $1 because of an Nameserver reset

command"

# use the ipmitool to send a power reset to <hostname>-ipmi

/usr/bin/ipmitool -H $1-ipmi -U <IPMI-User> -P <IPMI-Password> power reset

# If required add the required commands to release the NFS-Locks

# on the storage. Here an example for EMC VNX


# /opt/EMC/release_lock -y -f $1 <EMC Control-Station IP>

/bin/logger `whoami`" Release NFS locks of HANA Node $1"

rc=$?

exit $rc



Link the high-availability scripts to the shared high-availability (HA) directory under /usr/sap/<SID>/HDB<NR>/HA

(remember that the SAP HANA name server is responsible for resetting the failed node):

cishanar01:/ # mkdir /usr/sap/<SID>/HDB<NR>/HA

cishanar01:/ # chown <SID>adm:sapsys /usr/sap/<SID>/HDB<NR>/HA

cishanar01:/ # cd

cishanar01:/usr/sap/<SID>/HDB<NR>/HA # ln –s /hana/shared/scripts/ucs_ha_class.py

cishanar01:/usr/sap/<SID>/HDB<NR>/HA # ln –s

/hana/shared/scripts/ucs_ipmi_reset.sh

cishanar01:/usr/sap/<SID>/HDB<NR>/HA # chown <SID>adm:sapsys *

Link the HA directory to the correct SID and SID location (on all nodes):

cishanar01: # cd /usr/sap

cishanar01:/usr/sap # ln -s ./T01/HDB00/HA







Global.ini

The SAP HANA storage connector API provides a way to call a user procedure whenever the SAP HANA name

server triggers a node failover. The API requires the files mentioned in the preceding section. The procedure is run

on the SAP HANA master name server.

To activate the procedure if a node failover occurs, you must edit the global.ini file in <HANA install

directory>/<SID>/global/hdb/custom/config/ and add the following entry:

[Storage]

ha_provider = ucs_ha_class

cishana01: # cd /hanamnt/shared/<SID>/global/hdb/custom/config

cishana01:/hanamnt/shared/ANA/global/hdb/custom/config # ls -l

-rw-r----- 1 anaadm sapsys 90 Feb 15 11:22 global.ini

-rw-rw-r-- 1 anaadm sapsys 9560 Feb 15 11:23 hdbconfiguration_1

drwxr-x--- 3 anaadm sapsys 4096 Feb 15 11:22 lexicon


-rw-r--r-- 1 anaadm sapsys 128 Feb 15 12:34 nameserver.ini

cishana01:/hanamnt/shared/ANA/global/hdb/custom/config #

cishana01:/hanamnt/shared/ANA/global/hdb/custom/config # vi global.ini

[persistence]

basepath_datavolumes=/hanamnt/data/ANA

basepath_logvolumes=/hanamnt/log/ANA

[storage]

ha_provider = ucs_ha_class

ha_provider_path = /usr/sap/HA

Restart the SAP HANA database to activate the changes.

Check with your storage vendor to determine whether any other settings are required for the storage model you are

using.

Mount Options

To use NFS for SAP HANA data and log volumes, you need to change the mount options from the default Linux

settings. Here is an example of an fstab entry for NFSv4:

<NFS-Server>:/<NFS-Share> /hana/data/<SID>/mnt<NR> nfs4

rw,bg,vers=4,hard,rsize=65536,wsize=65536,nointr,actimeo=0,lock 0 0

Make sure that the <SID>adm user owns the data and log volumes; use the chown command after the file

systems are mounted.

Check with the storage vendor to learn what settings are required for the storage model you are using.

Block Storage for SAP HANA Shared File System

If you use block devices for the SAP HANA shared files in /hana/shared, you must configure the solution properly

to provide high availability for SAP HANA.

Oracle Cluster File System 2

To use Oracle Cluster File System 2 (OCFS2) for the SAP HANA shared file system, you need to install and

configure a cluster service.

Here is the link to the SUSE documentation:

https://www.suse.com/documentation/sle_ha/singlehtml/book_sleha/book_sleha.html - cha.ha.ocfs2.

Note: The size of the /hana/shared file system must be at least equal to the main memory of all SAP HANA

nodes, and the file-system type must be able to expand the size if a new node is added to the SID. To shrink file

systems on a block device or to shrink the block device itself, in most cases you need to delete and re-create the

file system or block device.

File Storage for /hana/shared

The SAP HANA data and log volumes are based on a shared-nothing model. In addition, SAP HANA requires a

shared area in which all SAP HANA nodes for one SID have access all the time in parallel: /hana/shared/<SID>.

https://www.suse.com/documentation/sle_ha/singlehtml/book_sleha/book_sleha.html#cha.ha.ocfs2


Here is a sample fstab entry:

<NFS-Server>:/<NFS-Share> /hana/shared nfs4

rw,bg,vers=4,hard,rsize=65536,wsize=65536,nointr,actimeo=0,nolock 0 0

Note: The size of the /hana/shared file system must be at least equal to the main memory of all SAP HANA

nodes, and the file-system type must be able to expand the size if a new node is added to the SID. One benefit of

NFS in most cases is the capability to shrink the file system on demand without any downtime.

Cisco UCS Solution for SAP HANA TDI Shared Network

This section describes the Cisco UCS Solution for SAP HANA TDI Shared Network implementation option. With

the introduction of SAP HANA TDI for shared networks, the Cisco solution can provide the benefits of an integrated

computer and network stack in combination with the programmability of Cisco UCS.

The SAP HANA TDI option enables organizations to run multiple SAP HANA production systems in one Cisco

solution, creating an appliance-like solution. Many customers already use this option for their nonproduction

systems. As another option, you can run the SAP application server using the SAP HANA database on the same

infrastructure as the SAP HANA database.

In addition to these two options, you can install a SAP HANA database in an existing Cisco UCS deployment used

to run SAP and non-SAP applications.

Multiple SAP HANA System IDs in One Appliance

With the SAP HANA TDI option, you can run multiple SAP HANA systems in the same infrastructure solution

(Figure 159). In this configuration, the existing blade servers used by different SAP HANA systems share the same

network infrastructure and storage systems. For example, in a solution with 16 servers and 4 storage resources,

you can run one SAP HANA system in an 8 + 1 configuration and another system in a 6 + 1 configuration, or any

other combination of scale-out and scale-up systems.


Figure 159. SAP HANA TDI: Two SAP HANA SIDs in One Appliance

Requirements

To use multiple SAP HANA SIDs in one appliance, one file system for /hana/shared per SID is required. For Fibre

Channel–based solutions, you should change the LUN mapping so that only the servers for a specific SID can see

the data and log LUNs.

Additional Options

Additional options include dedicated VLAN IDs per SID and QoS settings per VLAN.


SAP HANA and SAP Application Server in One Appliance

You can run the SAP HANA–related SAP application server on the same infrastructure as the SAP HANA

database (Figure 160). With this configuration, the solution controls the communication between the application

server and the database. This approach quarantines the bandwidth and latency for best performance and includes

the application server in the disaster-tolerant solution together with the SAP HANA database.

Figure 160. SAP HANA TDI: Database and Application in One Appliance


Requirements

To use SAP HANA and the SAP application server in one appliance, a dedicated server for the SAP applications is

required. You can use the same server type as for the SAP HANA database (Cisco UCS B440 M2), or you can add

servers such as the Cisco UCS B200 M3 Blade Server, to run the application directly on the blade or as a

virtualized system with a supported hypervisor.

The storage for the OS and application can be hosted on the same external storage as used for SAP HANA.

However, this setup can degrade the performance of the SAP HANA databases on this storage system, so

separate storage is recommended.

Additional Options

Additional options include dedicated VLAN IDs and QoS settings per VLAN. You can introduce a dedicated

application to the database network based on VLAN separation.

SAP HANA in an Existing Cisco UCS Deployment

You can deploy SAP HANA in an existing Cisco UCS deployment (Figure 161). In the example here, a FlexPod

solution is used to show how this option works. The FlexPod infrastructure is built from the same components as

the Cisco and NetApp solution for SAP HANA. Therefore, you can run one or more SAP HANA instances in a

FlexPod system by following these rules:

● Use only certified servers for SAP HANA.

● The network bandwidth per server must meet the guidelines described earlier.

● The storage must pass the test tool for SAP HANA TDI shared storage.


Figure 161. SAP HANA TDI: SAP HANA in a FlexPod Solution

Requirements

To deploy SAP HANA in an existing Cisco UCS deployment, a dedicated server for SAP HANA is required, with 10

Gigabit Ethernet for each node for SAP HANA internal traffic.

The storage for the OS and SAP HANA can be hosted on the same external storage resource as for all other

applications, as long the KPIs for SAP HANA TDI shared storage are met. However, separate storage is

recommended.

Additional Options

Additional options include dedicated VLAN IDs and QoS settings per VLAN.


Conclusion

You can connect external storage to Cisco UCS in a wide variety of ways. The long-standing industry best

practices for Fibre Channel, FCoE, iSCSI, NFS, and Common Internet File System (CIFS) apply with no changes

for Cisco UCS unless you want to deploy a direct-connect topology with Cisco UCS and the storage. Using both

NetApp and Cisco UCS with best practices for both results in reliable, flexible, and scalable computing and storage

infrastructure.

For More Information

● Cisco UCS Manager GUI configuration guides

● Cisco UCS B-Series OS installation guides

● SAP HANA at Cisco.com

● Central SAP HANA webpage

● Central SAP HANA support webpage

http://www.cisco.com/en/US/partner/products/ps10281/products_installation_and_configuration_guides_list.html

http://www.cisco.com/en/US/products/ps10280/products_installation_and_configuration_guides_list.html

http://www.cisco.com/c/en/us/solutions/data-center-virtualization/high-performance-analytic-appliance/index.html

http://www.saphana.com/docs/DOC-1067

http://service.sap.com/hana


Appendix A: Intel Westmere–Based Cisco Solution for SAP HANA Scale-Out

This section provides an overview of solution design for an appliance-like SAP HANA scale-out implementation.

The initial solution design was developed to meet all the requirements from SAP for a SAP HANA scale-out

appliance. The term “appliance” is somewhat misleading because this implementation is a preinstalled solution for

SAP HANA and not an appliance. SAP HANA 1.0 up to Service Pack 6 required all hardware components, such as

server, network, and storage resources, to be dedicated to one single SAP HANA database.

The solution design developed by Cisco for SAP HANA uses the following components:

● Cisco UCS

◦ Two Cisco UCS 6248UP 48-Port or 6296UP 96-Port Fabric Interconnects

◦ One to five Cisco UCS 5108 Blade Server Chassis

◦ Two Cisco UCS 2204 Fabric Extenders connected with four 10 Gigabit Ethernet interfaces to fabric

interconnect

◦ Three to 16 Cisco UCS B440 M2 High-Performance Blade Servers with two Cisco UCS VIC 1280 cards

◦ Two 10 Gigabit Ethernet uplinks for each server for dedicated bandwidth

● Cisco switches

◦ Two Cisco Nexus 5548 or 5596 platform switches for 10 Gigabit Ethernet and Fibre Channel connectivity

◦ Two Cisco Nexus 2224TP GE Fabric Extenders to connect devices that do not support 10 Gigabit

Ethernet

● Storage

◦ EMC VNX5300 unified storage, with NFS for the OS and Fibre Channel for data files and log files

◦ NetApp FAS3240 and FAS3250, with NFS for the OS, data files, and log files

● Cisco UCS C-Series Rack Servers

◦ Two Cisco UCS C220 M3 Rack Servers to host the management virtual machines

◦ One SLES-based management server as a VMware virtual machine

◦ One Microsoft Windows–based monitoring server as a VMware virtual machine

● Cisco 2911 Integrated Services Router (ISR)

◦ Serial console access to all devices

◦ Optional Network Address Translation (NAT) configuration for Cisco Smart Call Home, Simple Network

Management Protocol (SNMP), and syslog

Note: The Cisco UCS C-Series servers, the management virtual machines, and the Cisco 2911 ISR are not part

of the validation and certification effort discussed here and do not have to implement SAP restrictions on changes.

Every VMware ESX version that is supported by the Cisco UCS C220 M3 can be installed. The Cisco UCS C220

M3 also can be integrated into a VMware vCenter domain. An NFS share can be mounted as a data store, and

files can be moved from internal storage to external storage. The Cisco 2911 ISR can also be used for VPN access

by the Cisco Technical Assistance Center (TAC).


Because the VLAN groups feature was not available in Cisco UCS at the time the solution was designed, with

releases prior to Cisco UCS Manager Release 2.1, all LAN and SAN traffic had to be forwarded to the Cisco

Nexus 5500 platform switches. The integration into the data center network was performed using access ports on

the Cisco Nexus 5500 platform switches, causing some discussion about how the integration should be

accomplished.

Because of the limited capabilities of SLES for handling multipath SAN devices, especially for a SAP HANA

disaster-tolerant implementation, PXE boot was used with the root file system on NFS. All required components

are available, with DHCP and TFTP on the management server and NFS shares on the storage systems (Figure

162).

Figure 162. Cisco Solution for SAP HANA Scale-Out Design

The operating systems for the SAP HANA nodes are located on an NFS share on the external storage and

mounted on the management server for easier administration. Cisco provides golden images to deploy the

operating systems for the SAP HANA nodes. The golden images are based on the operating systems used in the

lab for testing and validation.


Cisco UCS configurations are available specific to the first-generation solution design and based on the Cisco UCS

B440 M2 High-Performance Blade Server.


Server-Pool Policy Qualification

The configuration of a server to run SAP HANA is well defined by SAP. In Cisco UCS, you can specify a policy to

collect all servers for SAP HANA in a pool. The definition in Figure 163 specifies all servers with 512 GB of memory

and 40 cores running at a frequency of 2300 MHz or higher.

Figure 163. Server-Pool Policy Qualification HANA-512GB-4870


The processor architecture is not of interest because the combination of 40 cores with 2300 MHz or more applies

only to the Intel Xeon processor E7-4870–based 4-socket server (Figure 164).

Figure 164. CPU Qualification Properties

The capacity is defined as exactly 512 GB of memory (Figure 165).

Figure 165. Memory Qualification Properties

Figure 166 shows the server pool for all the servers.


Figure 166. Server Pool

Server-Pool Policy

The server pool for the SAP HANA nodes and the qualification policy are also defined. In this case, the two

definitions are mapped together (Figure 167).

Figure 167. Server-Pool Policy HANA-512GB-4870

As a result, all servers with the specified qualification are now available in the server pool (Figure 168).

Figure 168. List of Servers in Server Pool HANA-512GB-4870


Adapter Policy Configuration

Figure 169 shows newly created Ethernet adapter policy Linux-B440 with values for RSS, receive queues, and

interrupts defined. This policy must be used for the SAP HANA internal network to provide the best network

performance with SLES 11.

Figure 169. Adapter Policy Linux-B440

Network Configuration

The core requirements from SAP for SAP HANA are met by Cisco UCS defaults. Cisco UCS is based on 10

Gigabit Ethernet and provides redundancy through the dual-fabric concept (Figure 170).


Figure 170. Network Paths with Cisco UCS

Each Cisco UCS chassis is linked through four 10 Gigabit Ethernet connections to each Cisco UCS fabric

interconnect. Those southbound connections can be configured in Port Channel mode or pinning mode. For the

preconfigured solution, the pinning mode was used for better control of the network traffic. The service profile

configuration helps ensure that through normal operation, all traffic in the internal zone is on fabric A, and all other

traffic (client-zone traffic and storage-zone traffic) is on fabric B. The management traffic is also on fabric A. This

configuration helps ensure that the network traffic is distributed across both fabrics.

With this configuration, the internode traffic flows only from the blade to the fabric interconnect and back to the

blade. All other traffic must travel over the Cisco Nexus 5500 platform switches to the storage resource or to the

data center network.

Because the Cisco solution with EMC storage uses Fibre Channel for the data and log volumes, two HBAs also

must be configured: one per fabric. The multipath driver for SLES is used to provide path redundancy and to

distribute the traffic over both fabrics as required. With integrated algorithms for bandwidth allocation and QoS,

Cisco UCS and the Cisco Nexus switches help provide the best traffic distribution.

LAN Tab Configuration

In Cisco UCS, all network types for a SAP HANA system are defined in VLANs. In the first-generation Cisco

solution, VLAN definition included only four networks: Admin, Access, NFS-IPv6 (for SAP HANA data and log

volumes), and Storage (for the NFS root). To run multiple SAP HANA systems, four areas (system IDs [SIDs]) were

predefined: T01 to T04 (Figure 171).


Figure 171. VLAN Definition in Cisco UCS

Each VLAN is mapped to a vNIC template to specify the characteristics of a specific network. The vNIC template

configuration includes settings such as MTU size, failover capabilities, and MAC address pools.

In the first version of the Cisco solution for SAP HANA, only four vNICs are defined (Figure 172).

Figure 172. vNIC Templates

The default setup is configured so that for most SAP HANA use cases the network traffic is well distributed across

the two fabrics (fabric A and fabric B). In special cases, this distribution may need to be rebalanced for better

overall performance. This rebalancing can be performed in the vNIC template with the Fabric ID setting. Be sure


that the MTU setting matches the configuration in your data center. For the internal vNIC, always set the MTU

value to 9000 for the best performance (Figure 173).

Figure 173. vNIC Template Details


SAN Access Configuration

For SAN access, you must configure vHBAs. A best practice is to configure vHBA templates: one for fabric A and

one for fabric B (Figure 174).



Now that all LAN and SAN access configurations and all policies relevant to SAP HANA are defined, a service

profile template can be configured.


On the General tab, select the UUID, server pool, and maintenance policy (Figure 175).

Figure 175. Service Profile Template: General Tab


On the Storage tab, define the WWNN configuration and the vHBAs based on the vHBA templates (Figure 176).


On the Network tab, define the required vNICs based on the vNIC templates. To map the pinning at the IOM layer,

you must manually configure the vNIC and vHBA placement (Figure 177).


In the Actions pane, select Modify vNIC/vHBA Placement. For the service profile template used for all servers in

slots 1 and 3, assign all vNICs to vCON1 (Figure 178).


Figure 178. vNIC and vHBA Placement for Slots 1 and 3

For the service profile template used for all servers in slots 5 and 7, assign all vNICs to vCON2 (Figure 179).

Figure 179. vNIC and vHBA Placement for Slots 5 and 7

On the Boot Order tab, use PXE-Boot for the Cisco solution (Figure 180). The starter kit variation of the solution

with a EMC block-only array as storage (using Fibre Channel for data and log files and OCFS2 for SAP HANA

shared storage) uses the SAN Boot option.


Figure 180. Service Profile Template: Boot-Order Sample for PXE-Boot


On the Policies tab, all the definitions already created are selected (Figures 181 and 182).



In the Cisco solution, nothing is configured on the iSCSI vNICs tab.


Service Profile Deployment

The basic settings are defined and captured in the service profile template for SAP HANA nodes. To deploy new

SAP HANA nodes, the only step that is required in Cisco UCS is to create service profiles from the specified

service profile template (Figure 183).

Figure 183. Service Profile Template: Before Service Profiles Are Created

Click Create Service Profile from Template, and a new window will appear. In this window, specify the service

profile name prefix, the starting number, and the number of service profiles that you want to create (Figure 184).

Figure 184. Creating a Service Profile from a Template


The specified number of service profiles will be deployed and mapped to a blade if one is physically available

(Figure 185).

Figure 185. Defined Service Profiles


The configuration details for every service profile now follow the definition that was created previously in the

service profile template. No intervention is required, and every service profile is ready for SAP HANA (Figure 186).

Figure 186. Service Profile Details for Service Profile Template

All vHBAs defined in the service profile template are configured with a WWPN from the pool (Figure 187).

Figure 187. Service Profile: Storage Tab


All vNICs defined in the service profile template are configured with a MAC address from the pools (Figure 188).

Figure 188. Service Profile: Network Tab

All policies are applied to the service profiles (Figure 189).

Figure 189. Service Profile: Policies Tab


Appendix B: Direct-Attached NFS Failure Scenarios

This section presents failure scenarios using NetApp FAS storage in the examples. Check with your storage

vendor to learn how to configure network failover for your storage resources.

Network-Attached Storage and Cisco UCS Appliance Ports

When you use NFS or CIFS with appliance ports, you must have upstream Layer 2 Ethernet switching to allow

traffic to flow in certain failure scenarios. This requirement does not apply to iSCSI-only traffic because the host

multipath I/O (MPIO) stack manages path failures and recovery end to end.

Appliance ports were introduced in Cisco UCS Release 1.4 and were designed to allow direct connection between

the Cisco UCS fabric interconnect and the NetApp storage controller. An appliance port is essentially a server port

that also performs MAC address learning. Given that the appliance port is a server port, the same uplink and

border port policies apply to them. They have border ports associated with them, and a network-control policy

determines what happens if the last available border or uplink port fails.

Appliance ports, like server ports, have uplink or border ports assigned through either static or dynamic pinning. By

default, if the last uplink port fails, the appliance port is taken down. You can change the network-control policy for

the appliance to make this event trigger a warning only. For NFS configurations, you should use the default setting,

which takes down the appliance port if the last uplink port fails. This setting helps ensure more deterministic

failover in the topology.

Cisco UCS fabric interconnects cannot run on vPC peers, so Cisco UCS fabric interconnects have an active-

passive data path for NFS traffic to the same IP address. You can, of course, perform active-active I/O processing

from both fabric interconnects to different back-end NetApp volumes and controllers.

You should use the interface group (ifgrp) feature to manage Ethernet ports on the NetApp controller. This feature

is called the virtual interface (VIF) in earlier releases of NetApp Data ONTAP. Both single-mode ifgrps and

multimode ifgrps are available. Single-mode ifgrps are active-standby interfaces. Multimode ifgrps can be static or

dynamic, and they support LACP port channels. For a detailed description of this feature, refer to the NetApp Data

ONTAP Network Management Guide.

Figure 190 shows the best-practice topology for Cisco UCS appliance ports with NFS or CIFS traffic. This

document uses this steady-state reference topology to examine various failure cases.

When using appliance ports in such a configuration, note these principles:

● The second-level ifgrp provides additional grouping of multiple multimode ifgrps, and it provides a standby

path if the primary multimode ifgrp fails. Because the primary multimode ifgrp is constructed using an LACP

Port Channel, such a complete failure is unlikely.

● Failure processing in Cisco UCS and in the NetApp controllers is not coordinated. Therefore, what one tier

sees as a failure, the other tier may not. This characteristic is the fundamental reason for the upstream

Layer 2 switching infrastructure shown in the figures that follow.

https://library.netapp.com/ecm/ecm_download_file/ECMP1155586

https://library.netapp.com/ecm/ecm_download_file/ECMP1155586


● NetApp ifgrp failover is based on the link state only. Therefore, if an event in Cisco UCS does not cause a

link-failure notice to be sent to the NetApp controller, then no failure processing will occur on the NetApp

controller. Conversely, NetApp interface group port migration may not cause any failures on the Cisco UCS

tier that trigger vNIC migration.

The remainder of this appendix presents several failure scenarios that illustrate these principles. Figure 190 shows

the steady-state traffic flow between Cisco UCS and the NetApp system. The red arrows show the traffic path from

the Cisco UCS vNICs to the NetApp exported volumes.

Figure 190. Steady-State Direct-Connect Appliance Ports with NFS

Failure Scenario 1: Cisco UCS Fabric Interconnect Fails (Simple Failure Case)

In Figure 191, the Cisco UCS fabric interconnect on the left fails. Cisco UCS sees this failure as a situation in which

the vNIC is moved from the A side to the B side by the fabric failover mechanism. No problems are encountered in

this scenario because the traffic fails over to the standby VIF. NetApp sees the fabric-interconnect failure as a link-

down event on the primary interface and thus expects to start seeing traffic on the standby interface and accepting

it.


Figure 191. Failure Scenario 1: Fabric Interconnect Is Lost

To recover from this failure scenario, the fabric interconnect needs to be repaired and rebooted.

When the fabric interconnect on the left comes back online, the traffic will not automatically return to a steady state

unless you use the NetApp Favor option when configuring the VIFs. Use of this option is considered a best

practice, and you should always implement it. If you don’t use this option, then the traffic will flow through the

upstream Layer 2 infrastructure as shown in Figure 192. Recall that the appliance ports perform MAC address

learning and have associated uplink ports—the reason that this traffic pattern is able to flow.


Figure 192. Data Path Upon Recovery of Fabric Interconnect

Failure Scenario 2: Cisco UCS Failure of Chassis IOM or All Uplinks to Fabric Interconnect

Failure scenario 2 is a complex and confusing failure condition. Cisco UCS sees this condition as an event for

which it should move the vNIC to fabric B; however, this condition does not appear as a link-down event even from

the perspective of NetApp. Thus, in this scenario, to allow traffic to continue to flow, an upstream Layer 2 device

must be present, as shown in Figure 193. Traffic will not be able to reach the NetApp volumes without using the

upstream Layer 2 network.


Figure 193. Failure Scenario 2: Loss of IOM

A reasonable question to ask is this: If you need to have upstream Layer 2 devices anyway, then why use

appliance ports in the first place? The answer is that the upstream device is used only in this one rare failure

scenario. The remainder of the time, your storage traffic goes directly from Cisco UCS to the array.

When the IOM or links in Figure 193 are restored, traffic flows normally as in the steady state because the NetApp

controller never made any changes in the primary interface.

Failure Scenario 3: Underlying Multimode VIF Failure (Appliance Port)

The failure scenario in Figure 194 is a case in which the appliance port link or Port Channel fails. NetApp sees this

condition as a link-down event and now expects to see traffic on the standby link. However, Cisco UCS does not

see this condition as a link-down event for the vNIC and thus keeps the vNIC assigned to fabric A. The uplink port

on the fabric interconnect enables the traffic to exit the fabric interconnect and then flow to the upstream Layer 2

network, and then back to fabric B, and then to the standby VIF on the NetApp controller.


Figure 194. Failure Scenario 3: Loss of Appliance-Port Links

Recovery from this failure scenario is similar to the recovery process discussed previously. Be sure to use the

NetApp Favor option in the VIF configuration to allow the steady state to return.


Failure Scenario 4: Last Uplink on Cisco UCS Fabric Interconnect Fails

As of Cisco UCS Release 1.4(3), this failure scenario is identical to the failure of a fabric interconnect as shown in

failure scenario 1. The default policy is to bring down the appliance ports if the last uplink is down (Figure 195).

Figure 195. Failure Scenario 4: Loss of Last Uplink Port on Fabric Interconnect


Failure Scenario 5: NetApp Controller Failure

Figure 196 shows the case in which one of the two NetApp controllers fails. A NetApp controller failover (CFO)

event occurs, assigning ownership of the volumes to the remaining controller. However, from the perspective of

Cisco UCS, nothing happened, so the vNIC remains on fabric interconnect A. Again, the only way that traffic can

continue to flow is from the upstream Layer 2 network because the failures are not coordinated.

Figure 196. Failure Scenario 5: Loss of NetApp Controller

Recovery from this event is a manual process. You must initiate the controller giveback command to return

everything to the steady state.


Summary of Failure-Recovery Principles

Here is a summary of the main principles for failure recovery when using appliance ports for NFS traffic with

NetApp arrays:

● You must have Layer 2 Ethernet upstream for failure cases. You cannot just deploy a storage array and

Cisco UCS hardware.

● Always enable an IP storage VLAN on the fabric interconnect uplinks for data-traffic failover scenarios.

● Provision the uplink bandwidth to accommodate the various failure cases discussed here.

● A minimum of two uplink ports per fabric interconnect is required. A port channel is an excellent

configuration.

● Do not deploy 1 Gigabit Ethernet uplinks with IP storage because you may experience a sudden

performance decrease during a failure event, unless this scenario is acceptable to the user community.

● At least a pair of 10 Gigabit Ethernet links per multimode VIF is recommended.

Printed in USA C11-733582-01 06/16

sap hana on cisco ucs: installation options

Documents