m5-32_server_overview_wbt 03-14-13.pdf

M5-32 Server Overview

Welcome to the “M5-32 Server Overview” Training.

Schedule: Timing Topic

57 minutes Audio

15 minutes Practice

70 minutes Total

Course Objectives

M5-32 Server Overview 1 - 2

M5-32 Server Introduction

The M5-32 Server is a Datacenter server that was designed with the highest reliability

and availability features. It has a high redundancy of components and performs

numerous health checks.


M5-32 Server Overview

The M5-32 server has very fast CMT (Chip-level Multi-threading) processors with a lot

of memory and the latest industry standard I/O.

As with previous servers, the M5-32 is modular with shared common components for

investment protection. The domaining of the server is highly flexible with hard partitions,

soft partitions, and application containers resulting in an industry leading virtualization

strategy.

The M5-32 Server features include application acceleration which means the M5-32

Server increases application performance and user response times. It uses six core M5

processors, high performance interconnections and industry standard PCIe Gen 3 I/O.

The M5-32 Server include State-of-the-Art RAS (Reliability, Availability and

Serviceability) features.


SPARC M5-32 Server

The M5-32 Server is based on the SPARC M5 processor and designed with emphasis

on core thread performance. The 6 S3 M5 CPU processor cores run at 3.6 Ghz

(gigahertz) with each core having its own dedicated 128K L2 cache. It runs as a single

or multi-threaded operation with up to 8 threads per core. This means one CPU with 6

cores can run 6 threads minimum and up to 48 threads maximum. The M5-32 Server

using its maximum CPU capacity has a total of 192 cores. (Maximum CPU core

calculation: 6 cores * 32 CPU core processors = 192 cores)

The 6 CPU processor cores can communicate between themselves at a very fast speed

since they share 48MB L3 cache (with four 12 MB L3 16-way cache) via a crossbar.

These features give the M5 CPUs high throughput with multi-threading and excellent

single thread performance.

On the right of the slide, you see a photo of the SPARC M5 processor.


SPARC M5-32 Server Make-Up

The M5-32 Server has the best features of the T-series servers with the high throughput

S3 SPARC processor core, Oracle VM Server for SPARC (LDOMs) virtualization and

ILOM system management. M-series servers are built on this extensive SPARC

heritage.

The M5-32 Server has the best features of the M-series servers with hardware

domains, flexibility and massive scalability since it is expandable to 32 processors and

32TB of system memory. It has high availability design, redundancy of all of its major

components and hot-plug replace or upgrade of these major components.

Maximum system memory calculation:

32 DIMM slots per CPU * 32-Gbyte capacity DIMMs = 1024 Gbytes per CPU

(2 CPUs per CMU * 1024 Gbytes = 2048 Gbytes per CMU) * 16 CMUs = 32,768 or 32

TB for M5-32 Server


M5-32 Server Comparison to M9000-32

The M5-32 Server is highly scalable and it has high availability providing the Data

Center a server with up to 32 CPU sockets and 1024 DDR3 DIMMs. Compared to the

M9000-32, the M5-32 Server provides six times better throughput performance. It has

three times more threads (1536) than M9000-32 with half the number of sockets (32). It

has four times more shared cache as the M9000-32. It has eight times the memory

bandwidth of M9000-32.


High-End Product Comparison

The M5-32 server most closely resembles the M9000-32. In this table you can see a

high-level comparison of the M5-32 Server to the M9000-32 server.

Processor - The M5-32 is a 3.5 GHz (gigahertz) chip multiprocessor while the M9000-

32 SPARC 6 VII (or 7) is 3.0 GHz (gigahertz) chip multiprocessor. There are 6 cores

per M5 processor so the M5-32 Server has a total of 192 cores (6 cores/CPU * 2 CMPs

per CMU * 16 CMUs maximum per M5-32 Server = 192 total core processors in the M5-

32 Server).

Memory – The total maximum system memory for the M9000-32 is 4 TB using 8GB

DIMMs * 512 DDR2 DIMM slots. The M5-32 Server has a total maximum system

memory of 32 TB using 32GB DIMMs * 1024 DDR3 DIMM slots.


Key Differences From Previous M-Series

In this table you can see how the M5-32 server differs from the previous M-series

servers.

SP - Administrators use XSCF (eXtended System Control Facility) running XCP (XSCF

Control Package) firmware for M8000/M9000 models to oversee their systems on-site

as well as from remote locations while they use the ILOM for the M5-32.

Domains - There are 24 hard domains (or Physical Domains) available for the

M8000/M9000 models while there are up to 4 Physical Domains (or PDoms) for the M5-

32 server along with up to 128 Logical Domains (or LDoms) per Pdom or up to 512

Logical Domains per M5-32 Server.

Memory - DDR3 is the system memory, or to get more specific, SDRAM (Synchronous

Dynamic Random Access Memory), used in the M5-32 with the older DDR2 in the

M8000/M9000 models. (SDRAM synchronizes the memory's responses to control

inputs with the system bus, allowing it to queue up one process while waiting for

another.)


Terminology

It is very important to understand the terminology used in the M5-32 Server before

going any further. Some of the terms may be familiar to you from other SPARC server

lines. Please take a minute to read the terms and their definition before continuing with

the training.


M5-32 Server – Front

Learn about the components the front of the M5-32 Server by engaging in this Student

Interaction.


M5-32 Server – Rear

Learn about the components the rear of the M5-32 Server by engaging in this Student

Interaction.


CMPs in CMUs

At the top of the slide, you see the CMU numbering as you look left to right at the rear

of the M5-32 Server. You see 16 CMUs numbers CMU 0 through CMU 15. At the

bottom of the slide, you see one CMU shadowed which is CMU 0 with an enlarged view.

The CMU contains memory and the two SPARC M5 processors in each CMP (Chip-

level multiprocessing or multicore processor) which are labeled CMP 0 and CMP 1.

Each SPARC M5 processor has 6 CPU cores and each core has L2 8-way 128K cache.


CMU Memory Components

Each CMU contains two memory boards with each memory board connecting to a CMP. The two memory boards are shown as light gray rectangles on the CMU diagram. There is one memory board for each CMP (Chip-level multiprocessing or multicore processor). The two CMPs are shown as two orange rectangles on the CMU diagram.

Each memory board has eight BoB (Buffer-on-Board) controllers. Four of the eight BoB controllers are connected directly to the CMP while the other four BoB controllers cascade into the first set of four BoBs. The four BoB controllers connected directly to the CMP are shown as light blue boxes on the diagram. The four cascading BoB controllers are shown as dark blue boxes in top of the diagram. Notice, BoB number one’s cascading connection to BoB number zero and BoB number zero connected directly to the CMP.

There are a total of 4 DIMMs for each Buffer-on-Board controller. These are Oracle Certified DDR3L (1.35V Low-voltage) DIMM modules that run at a data transfer rate of 1066 MHz. DIMM capacities that are supported are 16GB and 32GB. In a CMU, there are a total of 64 DIMM slots.


I/O Unit Components

On this slide you see the IOU. On the bottom left of the IOU, you see an I/O Board in

IOB 0. On the top left of the IOU, you see an I/O Board in IOB 1. The I/O boards are

required in order to access the internal HDDs/SSDs. Each I/O board provides the

connection to two EMS cards. The 10GbE interfaces are on the EMS cards.

In the center of the IOU, you see two cages with each cage containing four HDD/SSD

Drives. The left cage is cage 1 and the right cage is cage 2. The slots are labeled HDD

0 through HDD 3 (left to right) in cage 1 and the slots are labeled HDD 4 through HDD 7

in cage 2.

On the bottom right of the IOU, you see eight PCIe Gen3 cards in slots labeled PCIe 1

through PCIe 8. On the top right of the IOU, you see 8 PCIe cards in slots labeled PCIe

9 through PCIe 16.

Below and above the HDDs in the center of the IOU, you see four EMSs. Labeling slots

starts at the bottom with EMS 1 through EMS 4.


Domain Configuration Unit (DCU)

DCUs (Domain Configuration Units) are the hardware building blocks of Physical

Domains (PDoms). Each DCU includes two or four CMUs, one SPP, one IOU & eight

fans. A Physical Domain (PDom) can be made from one or up to four DCUs. Each

PDom operates like an independent server that has full hardware isolation from other

PDoms on the M5-32 Server.


Follow the instructions in this Student Interaction to become familiar with M5 CPU

processors use of Coherency Links.


Follow the instructions in this Student Interaction to become familiar with M5 CPU

processors use of Scalability Links.


CMUs Connected to Scalability Switch

In this diagram you see 4 DCUs which is the maximum that can be configured in a M5-32 Server. Each DCU has 4 CMUs which are connected to one of the 12 Scalability Switch Boards (SSBs). Each SSB has a Bixby (BX) ASIC (Application-Specific Integrated Circuit). The ASIC connects the CPU, memory, and I/O. The Bixby ASIC also controls the cache coherency and holds a portion of the Central Coherency Directory (reverse directory of all L3 caches on all the processors). In a directory-based coherence, the data being shared is placed in a central directory that maintains the coherence between caches. The directory acts as a filter through which the processor must ask permission to load an entry from the primary memory to its cache. When an entry is changed the directory, it either updates or invalidates the other caches with that entry. The ASIC determines who requested the data, where the data is located and where to send the data.

Each of the 12 Scalability Switch boards has a slot number labeled according to its position in the assembly. The first SSB slot is BX 0 and the last SSB slot is BX 11. Even numbered CMUs are connect to odd numbered SSB boards. For example, CMU 0 is connected to BX 1, BX 3, BX 5, BX 7, BX 9 & BX 11 ASICs (shown with black connecting lines). Odd numbered CMUs are connect to even numbered SSBs. For example, CMU 1 is connected to BX 0, BX 2, BX 4, BX 6, BX 8 & BX 10 ASICs (shown with red connecting lines).


Two CMU Connections to PCIe Slots

In the M5-32 Server, there is a relationship between a particular CMP’s pci port, the PCIe switch port in the I/O board and the PCIe slot in the IOU. The diagram on the slide depicts those relationships with its color coding. On the diagram, you see one DCU with two CMUs labeled CMU0 & CMU3. Each CMU has two CMPs labeled CMP0 and CMP1 with labeled pci slots (e.g. CMU0’s CMP0 has pci slots 0 and 1).

In the center of the diagram, you see two I/O Boards (IOB) labeled IOB 0 and IOB 1. Each I/O board has 4 PCIe switches. One switch in the group four switches are populated with connections from the CMP pci ports to the PCIe slots shown at the bottom of the diagram.

On the left side of the diagram, you see CMU 0 CMP 0 (shown in blue) pci port 0 is connected to IOB 0’s second switch from the left and then routed to PCIe slots 1 and 2. CMU 0 CMP 0 pci port 1 is connected to IOB 0’s leftmost switch and then routed to PCIe slots 3 and 4. You also see an EMS (Ethernet Module and Storage) connection from this switch to EMS 1. EMSs are PCI compliant connectors for PCI signals to be accepted by the EMS and for it to route SAS signals to the IOB Back-Plane. CMU 0 CMP 1 (shown in yellow) pci ports 2 & 3 are routed into the other two switches in IOB 0.


Four CMU Connections to PCIe Slots

On the diagram, you see one DCU with four CMUs labeled CMU0, CMU1, CMU2 &

CMU3. You see that there are only two PCIe slots connected to each CMP’s pci port

instead of four PCIe slots you saw used in the DCU configuration with only two CMUs

on the previous slide. While the previous example’s configuration provides the best I/O

availability, this DCU configuration with 4 CMUs provides the most redundancy.


Two Clock Boards

There are two clock boards in the M5-32 Server so one clock board is active (Clock 0)

and the other clock board (Clock 1) is on standby for failover in this redundant

configuration. Each clock board has a single clock source per board.

If the active clock board fails, the system to reboots and it uses the redundant clock

board and marks the failed clock board as inactive. The failed board can be replaced

while the system is running on the alternate clock board. An ILOM command is

required to be run to replace the failed clock board with a new one. After its

replacement, the new clock board is recognized and can now be used for failover if

needed.


Service Processor (SP)

The Service Processor is redundant in the M5-32. It provides the primary platform

configuration and management for the M5-32 Server. It works with the SPP (Service

Processor Proxies) in each DCU to configure and monitor the components in each

DCU. It runs the ILOM (Integrated Lights Out Management) software to manage the

M5-32 Server.

There are two service processors (SPs) in the M5-32 server. One acts as the primary

and the other as the standby to support automatic failover.


Service Processor Proxy (SPP)

A Service Processor Proxy (SPP) is a local service processor dedicated to a DCU. The

SPP’s purpose is to manage the CPU, memory controllers and DIMMs within a DCU.

Also, it does environmental monitoring of internal sensors (e.g. temperature sensors

and adjusts the 8 connected fans according to need). It also provides the rKVMS

(Remote Keyboard/Video/Mouse/Storage) console for remote management support of

the SPP (e.g. remote booting of physical & virtual servers after a firmware update).

In a M5-32 Server, there are four SPPs with each SPP being dedicated to one of the

four DCUs and commanded by the Main Service Processor (SP). Audit logs, faults,

etc. are forwarded from the SPPs to the Main-SP.

In a multi-DCU domain scenario, the SPP with the lowest number available in the

domain will become the Golden SPP that will manage the Physical Domain for all the

DCUs in the domain. The configuration from the Golden SPP is backed-up to the Main-

SP.


SP and SPP Connectivity

The SP and SPPs communicate via internal 100 Base-T Ethernet connections. The

100 in the media type refers to the transmission speed of 100 Mbit/s for the 100BASE-

T twisted pair Ethernet cables.

The SP manages the M5-32 Server chassis along with the 16 port Ethernet switch. The

Ethernet switch is a VLAN switch so it can be partitioned, etc. The SPPs are connect to

the Ethernet switch.

Each SPPs manage 8 CPUs (or 4 CMUs), monitor the 8 fans, send I/O signals to I/O

boards, manage the rKVMS activity, etc. If there is an issue with a fan, the SPP informs

the SP of any cooling issues and the SP adjusts fans if needed. If the fans are not

working for a CMU, there will be no airflow so the CMU will be shutdown.


rKVMS

The rKVMS (Remote Keyboard/Video/Mouse/Storage) is a virtual device that is part of the SPP. The virtual rKVMS do NOT exist on the SPs. The rKVMS provides a virtual USB keyboard, video, mouse and storage for the user.

The M5-32 Server supports both serial-redirection and video-redirection for each configured Physical Domain. However, the domain console is only supported via serial-redirection. The domain console is NOT supported via video-redirection. The video window allows a user to connect to the domain but it is only an X session connection. A user can launch JRC (Java Remote Console) on their laptop or server from the BUI on the Main-SP. The user selects the desired domain, selects either "serial-redirection" or "video-redirection' and then clicks the Launch button. JRC is loaded on the user’s local system and this initiates a connection to that domain's Golden-SPP. All communication via JRC and the ILOM is to the Golden-SPP for that domain. None of the rKVMS network traffic goes to the Main-SP. JRC is launched from the Main-SP but it connects to the Golden-SPP which 'is unique to the M5-32 Server. The Golden-SPP manages rKVMS the same as any single-SP ILOM system such as the T5 Server. On M5-32 Server, you can remove the I/O path to the rKVMS devices just like you can with the T5-8 where the CMUs are also removable.


Service Processor Software (ILOM)

The Oracle ILOM (Integrated Lights Out Management) software enables active

management and monitoring of the M5-32 Server. The ILOM on the M5-32 Server

looks and behaves just like ILOM on other platforms except is it also has a simple

(user-visible) set of extensions to support Enterprise features, such as, Physical

Domains, Service Processor Proxies and redundant Service Processors. These

extensions have a minimal impact on user experience.

The ILOM runs on the SP (Service Processor).


Oracle ILOM Key Functions

The Oracle ILOM is the system management firmware that is preinstalled on the M5-32

Server. ILOM functions provide the user the ability to perform firmware updates,

Remote Host Management, Inventory and Component Management, System

Monitoring and Alert/Fault Management, User Account Management and Power

Consumption Management. So, with all these functions, the ILOM enables you to

actively manage and monitor components installed in your M5-32 Server.

The Oracle ILOM SP runs independently of the server and regardless of the server

power state as long as AC power is connected to the server. When you connect the

server to AC power, the ILOM service processor immediately starts up and begins

monitoring the server. All environmental monitoring and control are handled by Oracle

ILOM.

Oracle ILOM provides a browser-based interface, a command-line interface, as well as

SNMP and IPMI interfaces. So, you see the ILOM is available through a variety of

interfaces. More information on the ILOM interfaces follows:

BUI interface - The web interface provides an easy to use BUI (Browser User Interface)


Advanced Virtualization Management

Advanced Virtualization Management provides a central interface for VM lifecycle management to provide quicker VM deployments and therefore increase productivity. This is accomplished via Solaris Zones, Oracle VM for SPARC & Dynamic Domains. The Customer is able to monitor VM or system-level utilization, reconfigure VM’s dynamically, create resource pools and migrate VM’s across servers.


Virtualization Software

The M5-32 Server has a high degree of virtualization including Physical Domains

(PDoms). The granularity of the Physical Domains is 8 CPU sockets (or 2 CMUs).

Logical Domains (LDoms) are supported within Physical Domains and Oracle Solaris

Zones for OS virtualization. Oracle Enterprise Manager Ops Center provides an

administrator a user-friendly interface for these different virtualization levels.


What is a Physical Domain?

There can be up to four physical domains (PDoms or PDomains) in the M5-32 Server.

Each PDom operates like an independent server that has full hardware isolation from

other PDoms in the chassis.

Domain configurable units (DCUs) are the building blocks of PDoms. Each PDom is

represented as /HOSTx in Oracle ILOM where x ranges from 0 to 3 (PDomain_0,

PDomain_1, PDomain_2, PDomain_3).

.


Two Types of Physical Domains

There are two Types of Physical Domains. The first type is an expandable unbound

Dynamic Domain or just a Pdom or Physical Domain. The second type is a Bounded

Dynamic Domain or Bounded PDom. The System Administrator sets the attribute of the

Physical Domains to select the type (e. g. attribute expandable=true).


Unbound Dynamic Domains

An expandable unbound Dynamic Domain or Physical Domain or Pdom allows each PDom to be different. A Physical Domain consisting of a single DCU is permitted to add new DCUs via a DR operation. In this way, the Physical Domain can expand the number of M5 CPU processors beyond its original 8 in its one DCU.

A Physical Domain greater than one DCU must use the SSB’s Bixby ASIC with its portion of the Coherency Directory to perform a lookup for remote DCUs so latency is higher for Dynamic Domains in comparison to Bounded Dynamic Domains. Each of the 6 Bixby ASICs hold 50% of the system directory (or Coherency Directory) for a M5-32 Server. Logically, the 12 SSBs are managed as two groups of 6. Each group of 6 can loose 1 SSB and it can run in a degraded mode of 5 and the system directory (or Coherency Directory) is still available.

One negative is that due to the system directory (or Coherency Directory) requirement, the loss of a Bixby ASIC causes a PDom reset.


Bounded Physical Domains

A Bounded Dynamic Domain (or Bounded PDom) does not use the Bixby ASIC’s

Coherency Directory for CMU to CMU access within the DCU. Because it does not use

the Bixby ASIC, it does less I/O and therefore has a lower latency compared to a

Dynamic Domain PDom. Another benefit to Bounded Dynamic Domains is that it is not

impacted by loss of SSBs.

The disadvantage of a Bounded Dynamic Domain is that is can never grow beyond a

DCU so it can not expand beyond 8 M5 CPU processors.


Physical Domain Examples

On the slide, you see nine PDom configuration examples for a M5-32 Server. Each

example uses 4 DCUs with each DCU configured with 4 CMUs. As the legend

indicates, the solid line around the DCU(s) represents a Bounded Dynamic Domain and

a dashed line around the DCU(s) represents an expandable unbound Dynamic Domain.

Starting with the three PDom configuration examples at the top of the slide, you see a

Dynamic Domain in the upper left example. PDom 0 has 4 DCUs configured as one

Dynamic Domain. The middle configuration shows a Bounded Dynamic Domain

labeled PDom 0 with one DCU and a Dynamic Domain labeled PDom 1 with three

DCUs. The rightmost configuration shows two Dynamic Domains labeled PDom 0 and

PDom 1. PDom 0 has one DCU and PDom 1 has three DCUs.

Take a minute to review the remaining six examples shown on the slide so you know

what Physical Domain configurations are possible within the M5-32 Server.


What Is a Logical Domain?

Logical Domains (or LDOMs) allow running multiple instances of Solaris on a single

physical platform. Logical Domains is the server virtualization and partitioning

technology based on SPARC V9 architecture. Oracle rebranded this technology as

Oracle VM Server for SPARC (after Oracle’s acquisition of Sun in January 2010). Each

domain is a virtualized environment with a reconfigurable subset of hardware resources

and an OS that can be started, stopped, and rebooted independently of the host system

or any other domains. Also, a running domain can be dynamically reconfigured to add

or remove CPUs, RAM, or I/O devices without requiring a reboot.


Hypervisor and Logical Domains

Hypervisor implements the software component of the virtual machine, providing low

overhead hardware abstraction. It enforces hardware and software resource access

restrictions for guests, including inter-guest communication, to provide isolation and

security. It also performs initial triage and correction of hardware errors.


What Is a Zone?

A zone is a virtual operating system abstraction that provides a protected environment

in which applications run. The applications are protected from each other to provide

software fault isolation. To ease the labor of managing multiple applications and their

environments, they coexist within one operating system instance, and are usually

managed as one entity.

The original operating environment, before any zones are created, is called the “global

zone” to distinguish it from “nonglobal” zones. The global zone is the operating system

instance.

Oracle Solaris Zones enable you to isolate one application from others on the same OS,

allowing you to create an isolated environment in which users can log in and do what

they want from inside an Oracle Solaris Zone without affecting anything outside that

zone.

Oracle Solaris Zones are secure from external attacks and internal malicious programs.

Each Oracle Solaris Zone contains a complete resource-controlled environment that

enables you to allocate resources such as CPU, memory, networking, and storage.


Oracle VM Server for SPARC: M5 Servers

Oracle VM Server for SPARC, previously called Sun Logical Domains, provides highly

efficient, enterprise-class virtualization capabilities for the M5-32 server.

A logical domain is a discrete logical grouping with its own operating systems,

resources, and identity within a single computer system. Application software can run in

logical domains. Each logical domain can be created, destroyed, reconfigured, and

rebooted independently.

Oracle VM Server for SPARC leverages the built-in hypervisor to subdivide system

resources (CPUs, memory, network, and storage) by creating partitions called logical

(or virtual) domains. Each logical domain can run an independent operating system.

Oracle VM Server for SPARC provides the flexibility to deploy multiple Oracle Solaris

operating systems simultaneously on a single platform.

This is the virtualization solution that fully optimizes Oracle Solaris and SPARC for your

enterprise server workloads.


M5-32 Server RAS by Hardware or Software Component

On this slide, you can see the many RAS features in each component of the M5-32

system. Take a minute to read the features listed for Oracle Solaris 11 FMA (Fault

Management Architecture), M5 processor, Hypervisor, central directory and switch,

Service Processor, Power and Cooling, System I/O and Memory components.

For example, Oracle Solaris 11 FMA (or S11 FMA) provides FRU Monitoring for hot-

upgradable components. The M5 processor has parity protection.

The M5 memory has extended ECC (Error-correcting code) error correction protection.

Hypervisor enables failure containment in the software partitions (Logical Domains).

The central directory and switch has CRC protection. There are two Service

Processors with automatic failover. The M5-32 system hardware has hot-plug CMU

(CPU Memory Unit) modules, disk drives, PCIe cards, Service Processor modules,

power supplies and fans. So, these are just a few examples of the RAS hardware

features for processor, memory, I/O, storage, power and cooling for the fault tolerant

M5-32 system design.


Optionally Download Training

Use the ATTACHMENT link in the upper

right-hand corner to optionally download

this training’s slides and notes in PDF

format.

Note: Student Interactions cannot be

viewed.


m5-32_server_overview_wbt 03-14-13.pdf

Documents