itanium® 2-based servers: superdome and mid-range€¦ · itanium® 2-based servers: superdome and...
TRANSCRIPT
Itanium® 2-based Servers:
Superdomeand mid-range
Mark ShawSr. Systems Architect, HP
Paul BouchierManagement Hardware Architect, HP
Rev B. 7/15/03Rev B. 7/15/03
HP World 2003 Solutions and Technology Conference & Expo page 211/20/2003
Agenda
Cellular systems– Chipset overview– System Architecture– High-Availability and reliability features– Security
System Management– Networked Service Processor/Console– Partition Configuration– Platform manageability HW/FW architecture– New security features
HP World 2003 Solutions and Technology Conference & Expo page 311/20/2003
Related Presentations
High Performance Systems Itanium Strategy #2594– Thurs 1:30
HP-UX Partitioning hands-on lab #2473– Fri 8:00
HP-UX Workload Manager hands-on lab #2474– Fri 10:10
HP World 2003 Solutions and Technology Conference & Expo page 411/20/2003
HP Cellular Systems FamilySuperdome
(PA-RISC & Itanium®)
PA-RISC: rp8420Itanium®
PA-RISC: rp7420Itanium®
2 cells, 8 cpu’s32 DIMM slots16 PCI slots,10U high, 19” rack
4 cells, 16 cpu’s64 DIMM slots32 PCI slots,17U high, 19” rack
16 cells, 64 cpu’s512 DIMM slots192 PCI slots
HP World 2003 Solutions and Technology Conference & Expo page 511/20/2003
sx1000 chipset includes five distinct chips
PCI/PCI-XBuses
Cell Controller
PCI-X System Bus Adapter
16 Links…2X MEMBUFFER
CPUSocket*
PCI-XHost
Bridge
32 SDRAMDIMMs
2X MEMBUFFER
2X MEMBUFFER
2X MEMBUFFER
Itanium® 2 Buses
* Accepts single or dual processor modules** Not used in 8 socket systems
CPUSocket*
CPUSocket*
CPUSocket*
Maximizes application performance from 4 to 128 processors
PCI-XHost
Bridge
Cell Board…
I/O Subsystem
Crossbar **
HP World 2003 Solutions and Technology Conference & Expo page 611/20/2003
System Performance and Scalability Features
Low-latency, high-bandwidth backplaneCell-based Design– Highly integrated cell controller with internal crossbar to
reduce latency and remove bottlenecks– Two processors (vs. four) attached to a single CPU front-
side bus to provide more bandwidth per processor and reduce queuing delays
– Large capacity, low-latency high-bandwidth memory– Memory directory coherency filters snoop traffic and
dramatically reduces load on the system backplaneDedicated Host Bridge ASIC and bus PCI-X slot– Performance and error isolation for each I/O card
HP World 2003 Solutions and Technology Conference & Expo page 711/20/2003
Cell and I/O assemblies
Integrated CPU / memory cell controller with internal crossbar– Low latency, high-bandwidth through a single ASIC– 2500 pins among the largest in the world– 32GB memory on 32 double-height DIMMs, 64GB coming– Four memory sub-systems
• Directory coherency– Four CPUs on two busses– Backplane and I/O links
I/O Controller and dedicatedPCI-X host bridge ASICs – 12 PCI-X Superdome slots– 8 PCI-X rx8620 & rx7620– Single PCI-slot per bus
HP World 2003 Solutions and Technology Conference & Expo page 811/20/2003
System Capacities
216128 GB8rx7xx0
42322256 GB16rx8xx0
448128 GB116Superdome SD16000
82962256 GB132Superdome SD32000
1621922512 GB164Superdome SD64000
nPartitionsI/O slotsMemoryCPUs
1 Memory capacities double next year
2 I/O expander required
HP World 2003 Solutions and Technology Conference & Expo page 911/20/2003
“Inventing” a dual core Itanium® CPU
The standard Intel cartridge packaging is not at maximum density– CPU silicon chip is in a package, on a carrier board, with power on the end– The basic chip and package could be packed much more densely
Industry-standard Itanium2® power pod (DC to DC power conversion)
Intel CPU chip inside package
Intel CPU “carrier” board
HP CPU “carrier” boardHP external cache andcontroller chip
HP power solution(goes on top rather than on the end)
HP World 2003 Solutions and Technology Conference & Expo page 1011/20/2003
Superdome architecture
C ELL 0
8.0 G B /s(72@ 500 M T/s)
2.5 GB/s (21@500MT/s diff)
4.0 GB /s(72@ 500M T/s) C ell C on troller
32 D IM Ms x 36 S DR A M4 G B /sec ( 288@ 125M T/se c)
C E L L
6.4 GB /s (144@ 400M T/s)
T otal Peak BW p er C ell: 12.8 G B /sec CP U s 16.0 G B /sec M e m 8 .0 G B /sec X bar 2 .5 G B /sec I /O
S ystem I/O : 8 P CI-X S yste m Bus A da pte rs : 96 s lots , no I /O exp bay 16 P CI-X S yste m Bus A da pte rs : 192 slots, req I/O exp bay
12 S lot I/O B ackplane4 @ 133MH z8 @ 66M Hz
C P U
C P U C P U
C P U
2X C rossb ar
11C E L L
9C E L L
10C E L L
8
I/OI/OI/OI/O
C E L L1
C E L L2
C E L L3
I/O I/O I/O
2X C rossb ar
C E L L4
I/O
2X C rossb ar 2X C rossb ar
C E L L5
I/O
C E L L6
I/O
C E L L7
I/O
C E L L12
I/O
C E L L13
I/O
C E L L14
I/O
C E L L15
I/O
PCI-X Host Bridge PCI-X 133System
BusAdapter2x
Mem
2x M
em2x
Mem
2x M
emB
uffe
rB
uffe
rB
uffe
rB
uffe
r
HP World 2003 Solutions and Technology Conference & Expo page 1111/20/2003
mid-range Itanium® architecture
CELL 0
8.0 GB/s(72@500 MT/s)
2.5 GB/s(20@500MT/s diff)
4.0 GB/s (72@500MT/s)
16 DIMMs x 36 SDRAM4 GB/sec (288@125MT/sec)
2x M
em
6.4 GB/s (144@400MT/s)
System I/O: 2 PCI-X System Bus Adapters: 16 PCI slots, embedded 4 PCI-X System Bus Adapters: 32 PCI slots, req I/O exp bay
Cell Controller8 Slot I/O Backplane
7 @ 133MHz1 @ 66MHz
CPU
CPUCPU
CPU
CELL1
CELL2
CELL3
I/O I/O I/O
2X Crossbar
PCI-X Host Bridge PCI-X 133System
BusAdapter
2x M
em2x
Mem
2x M
emB
uffe
rB
uffe
rB
uffe
rB
uffe
r
HP World 2003 Solutions and Technology Conference & Expo page 1211/20/2003
mid-range Itanium® architecture
CELL 0
8.0 GB/s(72@500 MT/s)
2.5 GB/s(20@500MT/s diff)
4.0 GB/s (72@500MT/s)
16 DIMMs x 36 SDRAM4 GB/sec(288@125MT/sec)
6.4 GB/s(144@400MT/s)
System I/O: 2 PCI-X System Bus Adapters: 16 PCI slots, embedded
Cell Controller8 Slot I/O Backplane
7 @ 133MHz1 @ 66MHz
CPU
CPUCPU
CPU
CELL1
I/O
PCI-X Host Bridge PCI-X 133System
BusAdapter2x
Mem
2x M
em2x
Mem
2x M
emB
uffe
rB
uffe
rB
uffe
rB
uffe
r
HP World 2003 Solutions and Technology Conference & Expo page 1311/20/2003
High availability featuresavailable across the family
Keep it running Fix it fast
diagnostic features:1. test station
- asic level scan tools- remotely accessible via lan
2. enhanced predictive support 3. high availability observatory4. ems monitoring system5. dynamic processor resilience6. dynamic memory resilience
n+1 features (hot swappable)1. cabinet blowers2. i/o fans3. dc power supplies4. cell backplane dc power
supplies
error correction1. ecc on cpu cache2. parity protected cpu & i/o
links3. single wire correction on
fabric & i/o4. ecc on all fabric and
memory paths5. Chip-kill memory
redundant ac input power(optional)
fault isolation technologies
online removal, replacement:1. cell assemblies*2. i/o cards
* note: os version dependent
HP World 2003 Solutions and Technology Conference & Expo page 1411/20/2003
Superdome Security Hardware
No path exists for an entity on one nPar to invade another
CorporatenPar
mem
mem
mem
mem
CPU
CPU
CPU
CPU
CellController
PCI-
XPCI-
X
I/O
nParfirewall mem
mem
mem
mem
CPU
CPU
CPU
CPU
CellController
PCI-
XPCI-
X
I/O
nParfirewall
CustomernPar
mem
mem
mem
mem
CPU
CPU
CPU
CPU
CellController
PCI-
XPCI-
X
I/O
nParfirewall mem
mem
mem
mem
CPU
CPU
CPU
CPU
CellController
PCI-
XPCI-
X
I/O
nParfirewall
Shared xbar fabric Hardw
are firewall enforced by CellC
ontroller nPar firewalls
Accesses (mem, I/O, etc) from one nPars toanother are caught and discarded at the
destination crashing only the nPar at fault
External WANCorporate LAN
Management Processor
No path to/from mgmt LANConsole, diagnostic & configuration bus is
not accessible by OS nor does itimplement any networking protocol
ManagementLAN
HP World 2003 Solutions and Technology Conference & Expo page 1511/20/2003
Agenda
Cellular systems– Chipset overview– System Architecture– High-Availability and reliability features– Security
System Management– Networked Service Processor/Console– Partition Configuration– Platform manageability HW/FW architecture– New security features
HP World 2003 Solutions and Technology Conference & Expo page 1611/20/2003
Platform Management
DefinitionsMP = GSP = Service ProcessorComplex: The entire box(es) including all partitions
Demo
HP World 2003 Solutions and Technology Conference & Expo page 1711/20/2003
SMS & SPOP configuration
SPOP
System/payload LANs
SMS
Management LAN
Internet
Customer Firewall
CMS
Data Center
Intelligent router or access server
CMS: Central Management Station (Service Control Manager, Insight Manager)SPOP: ISEE Support Point of PresenceSMS: Support Management Station
HP World 2003 Solutions and Technology Conference & Expo page 1811/20/2003
Partition Management
Intranet
SMS
Management LAN
CMS
Platform Platform ((SdomeSdome, , midrange)midrange)
MPMP
nParnPar
Intranet
Intelligent router or access server
P.M.P.M.ToolsTools
P.M.P.M.ToolsTools
Web Browser/Web Browser/Remote loginRemote login
P.M.P.M.ToolsTools
P.M Tools = partition management tools (parmgr GUI, parmodify etc. cmd line tools)
MP MP cmdcmdcan disablecan disablereconfigurationreconfigurationby partitionsby partitions
P.M. tools connect to MPP.M. tools display routing
HP World 2003 Solutions and Technology Conference & Expo page 1911/20/2003
Partition Management
P.M Tools run on nPAR:• Benefit: browser session can manage other aspects of OS (e.g. sam over web).
Parmgr can launch other apps (e.g. PCI OL*).
• Security consideration: requires npar to be enabled to reconfigure the complex.
P.M. Tools run on CMS• Benefit: Tools can be launched from data-center level admin tool. Browser
session can manage other aspects of OS (e.g. sam over web).
• Security consideration: requires npar to be enabled to reconfigure the complex.
P.M. Tools run on SMSBenefits: Partitions can be reconfigured in the absence of any running OS. Partitions can be disabled from reconfiguring partitions (security benefit). Usability Consideration: Partition management system may not have easy network access to the system for OS management. This may or may not be important to the customer.
HP World 2003 Solutions and Technology Conference & Expo page 2011/20/2003
Console and PC SMS Network ConfigurationSingle IPF or PA-8800 Superdome
CE Laptop
Customer Mgmt LAN(If Desired)
UnconnectedManagement Processor
Private LAN PortCustomer LAN Port
RS232 Local
RS232 Remote
Partition 1
Partition 0CIO LAN
CIO LAN
Superdome Server
PC SMS orCONSOLE DEVICE
LAN HUB
Modem
Optional**
Optional**VGA/USB
KVM SW
OptionalWindows Partitions Only
** Security concerns may dictate that a partition LAN not be connected to the management LAN.Alternatives: 1) Access from a management station to a partition LAN through a secure router. 2) Text mode access to commands via the console. *** Serial cable 24542G needed for initial install
HP World 2003 Solutions and Technology Conference & Expo page 2111/20/2003
Partition Manager New Features
New web interfaceGraphical “big picture” views of
• nPars• Hardware in Complex
Supports new OS/HW features• Cell local memory for HP-UX
11i v.2 partitions• Inter-partition security
Remote admin of Superdome Madison complexCompatible with iCOD/Pay-Per-UseIncreased integration with SCM 3.0Native on Windows (2H03)
J2EE app runs in tomcat web server
HP World 2003 Solutions and Technology Conference & Expo page 2211/20/2003
MP operation
Always remember – HE from the main menu gets HELP!
HP World 2003 Solutions and Technology Conference & Expo page 2311/20/2003
MP operation – Console menu
HP World 2003 Solutions and Technology Conference & Expo page 2411/20/2003
Telnet Clients for Console Access
MP
IPF FW
Linux
BestWindows
LimitedBestHPUX
Other ANSI or VT100
Hypertermor telnet from cmd.exe window
Reflection1 for HP or hpterm (X)
Console
HP World 2003 Solutions and Technology Conference & Expo page 2511/20/2003
MP operation - VFP
HP World 2003 Solutions and Technology Conference & Expo page 2611/20/2003
Command menu concepts
• Shared environment• all users in the command menu share the environment,• i.e. all users see commands typed by other users
• Command ownership• once a user types a command name, e.g. RS, they have
exclusive ownership of that command until the command completes or an inactivity timer expires.
• Command timeout• if a user types a command but doesn’t complete the
action before the timeout expires, the command will be aborted
HP World 2003 Solutions and Technology Conference & Expo page 2711/20/2003
Command menu concepts
• 3 User Capability Levels, set by SO command 1. Single_Partition user Can only connect to assigned partition Can only execute commands that affect their assigned
partition 2. Operator Has single partition user capabilities on all partitions Can’t reconfigure the MP 3. Administrator Has operator level capabilities Can reconfigure the MP
HP World 2003 Solutions and Technology Conference & Expo page 2811/20/2003
Boot Startup / Shutdown / Reset
RS (Reset) Command: Irrecoverably halts all the system processing and I/O activity and restarts the Partition chosen.
TC (TOC) Command: Irrecoverably halts all the system processing and I/O activity and dumps core then restarts the Partition chosen.
RR (Reset for Reconfiguration) Command: Irrecoverably halts all the system processing and I/O activity and restarts the chosen Partition in a way that it can be reconfigured. (Partition stops at BIB.)
BO (Boot) Command: BO releases a partition to boot that has been reset for reconfiguration, or that has failed to boot because of an invalid configuration. BO clears the "BIB" bit (Boot Is Blocked). RR and BO are natural pairs. However, it should not be necessary to use them because Parmgr is the preferred tool. But, like RS, they can get you unstuck if the system won't shut down or won't boot.
HP World 2003 Solutions and Technology Conference & Expo page 2911/20/2003
RS Command
MP:CM> rs Execution of this command irrecoverably halts all the system processing and I/O activity and restarts the Partition chosen. # Name --- ---- 0) Partition 0 1) Partition 1 Select a partition number : 0 Do you want to reset Partition number 0, named Partition 0? (Y/[N]) y
-> Partition selected will be reset.
HP World 2003 Solutions and Technology Conference & Expo page 3011/20/2003
Manageability Subsystem HW Architecture
PM FW
MP FW
CLU FW
PDHC FWPDHC FWPDHC FWPDHC FW
CIO FWCIO FWCIO FWCIO FW
UGUY board
Power EnableBackplane Reset
System FW, OS & Manage-mentApps
PDH Hardware
Cell Board and PDH Daughtercard
CPU ModuleCPU
Module
SM Bus
Power & Reset HW
SBC & SBCH bd
CIO & IOBP bds
Console DataRIO Cable
MP Bus (USB)
HP World 2003 Solutions and Technology Conference & Expo page 3111/20/2003
Superdome Partitions: What are they?
Partitions are physical entities defined around cell board boundaries A partition’s I/O consists of the I/O chassis attached to cells in the partition.A partition must contain at least one I/O chassis with a core I/O card along with sufficient I/O resources to boot the partition.An instance of an operating system uses the cells and I/O chassis assigned to the partition.Dynamic addition of new Partitions Adding/removing cell boards to existing Partitions does not impact any other PartitionFollow-on release will increase the dynamism.
HP World 2003 Solutions and Technology Conference & Expo page 3211/20/2003
Partitions on an eight cell SD32000
One 32-Way Partition
I/O
XBCI/OI/OI/O
CellCellCellCell
I/O
XBCI/OI/OI/O
CellCellCellCell
Two 16-Way Partitions
I/O
XBCI/OI/OI/O
CellCellCellCell
I/O
XBCI/OI/OI/O
CellCellCellCell
H/W FireWall
HP World 2003 Solutions and Technology Conference & Expo page 3311/20/2003
Complex profile conceptsComplex Profile is group of data structures which define:• SCCD - Stable Complex Config Data
• Global parameters (affecting all partitions):• Cell -> partition association• system name, serial #, etc
• PCD – Partition Config Data• Partition parameters (affecting one partition)
• Partition name• Active Cells
Complex Profile stored in MP (master copy) and on every cellComplex Profile initially created by the MP (CC command)
• Read and modified by the Partition Manager, the partition commands, and system firmware
HP World 2003 Solutions and Technology Conference & Expo page 3411/20/2003
Partition boot process
Cell reset is released after power up or RS commandSystem FW begins booting cell (not part of a partition yet)System FW sets the “shared memory good” flagPDHC and MP begin checking complex profile for consistency across cells. PDHC and MP may push out a modified complex profile.Concurrently, System FW does memory and I/O discoverySystem FW on a cell checks the “Boot Is Blocked” (BIB) flag and spins waiting for it to be clearedPDHC and MP clear the BIB flag after the complex profile has been checkedCell boots, reads the complex profile, and rendevous with the other cells in its partition.
HP World 2003 Solutions and Technology Conference & Expo page 3511/20/2003
Configuration Management System
PDHController
PDHController
Boot ROMBoot ROM
NVM
PDHGPE regsGPE regs
IPMIIPMI
CC ASICCC ASIC
CPUCPU CPUCPU CPUCPU CPUCPUCell HW
Cell HW
MemoryMemory
CabinetMonitoring &
config reporting
CabinetMonitoring &
config reporting
UGUY MPBus
MP CPUMP CPUNVMNVM
MP FWMP FWMP
Complex ProfileComplex Profile
Complex ProfileComplex ProfileCommand to MPCommand to MP
FWOSOS
Mgmt App
IPMIDrvr
1
1
1 Mgmt app sends new complex profile to MP2 MP distributes it to affected cells
2
3
3 FW & OS use it next boot or during OLA
PMIPMI
HP World 2003 Solutions and Technology Conference & Expo page 3611/20/2003
Event Management System
PDHController
PDHController
Boot ROMBoot ROM
NVM
PDHGPE regsGPE regs
IPMIIPMI
CC ASICCC ASIC
CPUCPU CPUCPU CPUCPU CPUCPUCell HW
Cell HW
MemoryMemory
CabinetMonitoring &
config reporting
CabinetMonitoring &
config reporting
UGUY MPBus
MP CPUMP CPUNVMNVM
MP FWMP FWMP
Event LogEvent Log
Command to MPCommand to MP
FWOSOS
Event Mgmt App
IPMIDrvr
PMIPMI
1
1
1 FW, platform or OS sends event to MP
1
2 Mgmt apps retrieve events from MP event log
2
22LAN to
SMS
HP World 2003 Solutions and Technology Conference & Expo page 3711/20/2003
New & Forthcoming Security Features
Parperm command, reconfigure disable, reflash disable– Enables admins to limit the capabilities of root on
partitionsIPMI password (SO command)– Enables admins to limit LAN user access to
reconfiguration capabilitiesSA command– Enables admins to disable services
Embedded Web Console– Provides privacy through encryption for MP/console
accessSNMP identification integrates with forthcoming Insight Mgr version
Interex, Encompass and HP bring you a powerful new HP World.