PowerPoint Presentation

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#1SPARC T5 Servers Deep Dive Part 2Insert Presenters Name HereInsert Presenters Title Here

0.20Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#2The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#ILOMCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Sun4vSun4vSun4vHostService ProcessorOBPOBPOBPHost FlashILOMLinux KernelPOSTHost ConfigCPUMemoryIOPlatform HWPILOT3 MicroprocessorHypervisorSolaris11U1SolarisS10U11SystemDomainGuest MgrEnvironmentalsFault ManagementLED ControlSP DiagsDFRUIDSPlat HW SvcFDD (diagnosis)IPMICLIsLogsSNMPFMA SupportPower On/OffFERGKernalFMA ComponentsPlatform DriversKernalFMA ComponentsPlatform DriversKernalFMA ComponentsPlatform DriversHost ConfigMachineDescriptionHypervisionOBPPOSTUBoot/DiagsOBP NVRAM/POST/SC config varsHostDataFlashASRDBLDOMS configConsole LogSER logTOD dataLDOMSManagerFPGASoftware/Firmware Block DiagramCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SolarisSolarisSolarisSolarisLDMPMHypervisorHCGMDDBPODfaultDBCAPIfddfmadmUIsControl DomainLogical DomainLogical DomainLogical DomainPhysical HardwareService ProcessorLegendCAPI ILOM Common APICOD Capacity on DemandDDB Deconfig DBfaultDB Fault DBfdd Fault Diagnosis DaemonGM Guest ManagerHC HostconfigLDM Logical Domain ManagerPM Power ManagerPOD Platform Obfuscation DaemonUI User InterfaceIntegrated T5 Software StackCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

T5 SPEmulex PILOT3 based SPSame SP card used on all T5 systemsSame SP card shared with M5 systems.Monitors voltages used on the SP via ADC in PILOT3Provides similar/same functionality as previous generations.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#7rKVMS (Java Remote Console+)rKVMS Solution provided by Emulex, optimized for Pilot3Expanded to support ILOM featuresILOM authenticationSerial Redirection (Host console redirection)New features:Auto detected Mouse modeVirtual keyboardStorage redirection: SSL and Non-SSLTake / Relinquish full controlLocal monitor on/offCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#8rKVMS (Java Remote Console+), con'tMore key differencesMouse and keyboard will only show up in the OBP device tree if remote console is active. Before, devices were always present.Video cannot be used as a system boot console. OBP rconsole alias removed.Front and rear VGA ports are distinct devicesConfigurable VGA_REAR_PORT policy under /SP/policyDefault disabled = front port is the port in useIf connecting to the rear VGA port, must change this setting.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#9Side-band Management3 Remote Management Communication ChannelsOut-of-band management = communicate with the SP over a dedicated media (Ethernet/Serial)In-band management = communicate with the SP through Oracle Solaris via agentsSide-band management = communicate with the SP over a shared media (the hosts data network interface)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#10

T5-2 ILOM Protocol InterfacesIn BandHardware Management PackSNMPAgent

CLI Tools

Out of BandSide BandSerialCLIHost Console RedirectionEthernetHTTPSRKVMSSSH (CLI)SNMP (traps)IPMI (syslog, SMTP, misc. IP Services)Serial over EthernetSide BandCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#1119/05/111111ILOM Offers a number of choices for physical connection: an onboard ethernet port for active and use, sideband access via the onboard host NICs.

In addition there are on board USB ports, and serial connections can be redirected over the LAN. Also physical keyboard, video, mouse can be attached.

T5-4/T5-8 ILOM Protocol InterfacesOut of BandIn BandSide BandSerialCLIHost Console RedirectionHardware Management PackSNMPAgent

CLI Tools

EthernetHTTPSRKVMSSSH (CLI)SNMP (traps)IPMI(syslog, SMTP, misc. IP Services)Serial over EthernetOut of BandCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#1219/05/111212ILOM Offers a number of choices for physical connection: an onboard ethernet port for active and use, sideband access via the onboard host NICs.

In addition there are on board USB ports, and serial connections can be redirected over the LAN. Also physical keyboard, video, mouse can be attached.SerialILOMSidebandHost OSCLIBrowserIPMISNMPrKVMSConsole RedirectionILOM Interface CapabilitiesCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#13While a server can use the CLI interface for basic management, it is highly recommended to attach ethernet to the ILOM in order to have a wider range to tools and interfaces to manage the server.

Oracle ILOM Key FunctionsManagement InterfacesCLI, BUI, IPMI, SNMPFirmware UpdatesRemote Host ManagementInventory and Component ManagementSystem Monitoring and Alert/Fault ManagementUser Account ManagementPower Consumption ManagementCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#14Guest Manager aka GMResides on the SPProvides services to Guest domains via Logical Domain Channels (LDCs)Communication bridge between Host and ILOMProvides FERG capabilitiesManages LDOM configurationsProvides development facilities like Configvars, eFuse etc.Can sequence the CPU in Serial boot mode for debug purposes.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#15HostconfigInitialization code that runs at power-on on SPARCPlatform specific code that drives initialization and configuration of CMP, memory and other motherboard components at power-onInvokes Power-On Self Test (POST) twice (socket, smp) and applies platform policies to configure system around failed componentsGenerates Physical Resource Inventory (PRI) based on FRU information and system configurationCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#16HostconfigHighly parallelized uses multiple strands to speed configurationMemory configured in parallelDeconfigures components in the deconfig db (DDB)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#17LogsConsole output (including Hostconfig) is captured in the console logs located on the SP at /persist/host_logs/ with the mapping:/HOSThostconsole.logVBSC/GM console logs are captured at /coredump/sp_trace/logs/GM.log.#, (log rotated files)

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#18Logs@(#)Hostconfig 1.3.x-nightly 2012/10/24 19:11 [t5-8:debug]2012-10-25 18:44:31 2:0:0> WARNING: TPM hardware is disabled2012-10-25 18:44:56 3:0:0> NOTICE: SPARC-T5 Revision 1.02012-10-25 18:44:56 0:0:0> NOTICE: SPARC-T5 Revision 1.0YYYY-MM-DD HH:MM:SS UTC timeSocket:Core:Strand reporting entityNOTICE: - general messageWARNING: - significant problem, doesnt inhibit boot ERROR: - signficant problem, may impact bootFATAL: - system cannot proceedDEBUG: - development use only, should never see theseCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#19PCIE fabric and failoverioreconfigure controls behaviorredundant paths to all IOonly a single path is active at a timefailover results in less available bandwidth per deviceCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#20Fault ManagementKnowledge Articles in MOSILOM fdd DiagnosisFaults and AlertsNo ALOM CompatibilityILOM FMA Captive ShellSideband Service Processor Network ConnectionNew ILOM Fault Notification (SNMP Trap)ASR SupportFMA on M5 ILOM also applies to T5 ILOM, except for M5 specific featuresCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#21Fault Management on T5 systemsT5 CPU and Memory faults are now diagnosed by ILOMFMA's Fault Proxy is used to keep ILOM's fault manager in sync with Solaris' fault manager. Both will display the sum of all faults in the system.Faults can be repaired from either side.Fault Proxy communicates via the Ethernet Over USB connection.IO faults are still diagnosed by Solaris.Disabled Database (DDB) owned by ILOMFor faults which diagnose resources as unusable, ILOM will add those resources to the DDB. Resources excluded on next host reset.When faults are repaired, ILOM automatically updates the DDB. Bringing components back online requires a host reset.Extended SP-POST (Power on Self Test)Runs at SP boot. Tests devices on the SP FRU and its Ethernet port.Status stored and converted to ereports after ILOM boots.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#22Fault proxyIO ereports are forwarded from the SP to the control domain, and then on to any relevant IO domainFaults are proxied between the SP, the control domain and any IO domains to provide a single view of faults in the system.Non-servicable faults such as memory faults are not proxied.The SP and the control domain can view and manage all faults in the system.An IO domain can only view and manage faults local to the domain.Control DomainIO DomainLDCLDCTCP/IPSPhostdFETDip-transprtETMETMETMETMETMip-transportereportsereportsfaultsfaultsLDCCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#23Ereport GenerationThree producers of ereports:Guest Manager (GM)Error Telemetry Collection Daemon (ETCD)Platform Obfuscation Daemon (POD)GM has direct communication with SW running on the host HWHostconfig (HC)POSTHypervisor (HV)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#HV Reported ErrorsCommunicates error information in a raw, binary format called a Service Error Report (SER)SERs are processed by a library called the FMA Ereport Generator (FERG)published to the Event Manager frameworkCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Platform Obfuscation Daemon (POD)Runs any POST functionality that does not involve SPARC codeEreports generated on HW problemsIf POD encounters HW problems not accessible to HC, an ereport is generatedCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#ASR SupportSPARC T5 servers will be supported by ASR (Automatic Service Request) at releaseContinues use of sunHwTrapFaultDiagnosed SNMP notificationTelemetry for ILOM fdd diagnosisSupports platform and FRU identitySupports multi-suspect listCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#27Service Processor Software (ILOM) on T5/M5 SystemsILOM looks/behaves just like ILOM on other platformsSimple (user-visible) set of extensions to support Physical DomainsExtensions to support Service Processor Proxies and redundant Service ProcessorsMinimal impact on user experienceCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Boot modeT-series platforms (except T5-1B) have two boot mode optionsSequenced Boot: SP boots, then user initiates host power-on via ILOMParallel Boot: SP and host power on in parallel to reduce overall boot timeAdjustable via ILOM '/SP/policy'-> show /SP/policy PARALLEL_BOOT = disabled

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#29Boot Sequence

Service Processor BootGrub starts on poweronGrub starts LinuxLinux starts various services, starts ILOMILOM starts Guest ManagerServices to the Guest OS domainsCommunication (via FPGA) bridge between the host and the ILOM serviceProvides Fault Error Report Generation (FERG)Manages LDom configurations in persistent storagePOD performs power sequencing for Host processorsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#30Boot Sequence (Continued)

Host BootGM takes processors out of resetT5 starts executing Hostconfig code from flashHostconfig code does the following:On each selects a master to coordinate the host configuration, runs other per-CMP strands in parallel for initialization and configurationAfter each overall host config is complete, populates PRI (physical resource index) and generates MDs for Hypervisor and GuestMaster strand jumps to Hypervisor; others strands are parkedCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#31Boot Sequence (Continued)

Host Boot (continued)Hypervisor then proceeds toCopy itself from ROM to RAMInitializes itself based on HV MD (mapping of phys resources to logical domains)Starts the guest (OpenBoot is the first guest)OpenBoot probes I/O devices based on Guest MD and sets up the device tree for Solaris. Starts Solaris boot.Boot block (bootblk) is loaded from diskbootblk reads UFS (or ZFS) file system to find ufsbootLoads ufsboot (or zfsboot) into low memory and jumps to itufsboot locates a kernel (Solaris), loads and jumps to itCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#32Boot Sequence (Continued)

Host Boot (continued)Solaris kernel runs and does the following to boot CPULoads required driversSets up the VM systemTakes over trap table (ufsboot disappears)Starts other CPUsHand crafts first process init and lets it run on all other CPUsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#33SP HighlightsILOM looks/behaves just like ILOM on other platformsSimple (user-visible) set of extensions to support Physical DomainsExtensions to support Service Processor Proxies and redundant Service Processors on M5-32Minimal impact on user experienceCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#ILOM on T5/M5 Enterprise FeaturesILOM extensions to support Enterprise FeaturesSome of these are conceptually leveraged from XSCF SWSP Tracing FacilityVery useful for tracing inter-process activityReliable performance (elapsed time) measurementsAllows for tracing interactions between SPs and SPPs etcEnterprise systems have lower volume, more complexconfigurations and high RAS expectationsWe cannot expect customers to reproduce bugsNeed to collect as much debug info as possible on live system as it occurs on customer siteCoredump compression and snapshot collectionUnified snapshot from all SPs and SPPsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#ILOM on T5/M5 Enterprise FeaturesConfstore distributed config databaseExploitable by CMM/BladesSystem Identity is maintained across FRU / SP replacementaka TLI (Top-Level-Identifier)Flash Images are signed to avoid compromised imageshardened edits of config filesTransaction oriented before or after, no intermediate resultsTunables framework for MAX_USERS etcSensor Broadcast (SSBCAST) enhanced to supportCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5-4 / T5-8 Processor ModulesILOM enforces valid PM / PFM configurationsT5-8 can be reduced to 6 or 4 processor configurations by replacing certain PMs with Processor Filler Modules (PFMs).Similarly, T5-4 can be reduced to a 2 processor config using a PFM.ILOM will create a configuration fault and refuse power-on. Must correct the configuration, repair the fault, and try again.Processor Modules can be dynamically added to a running system Post-RR, that is...New form of POST: iPostWill test a PM that is powered-on with DR.ILOM part of iPost tests all i2c devices on the PM.Processor Module DR will be accomplished through ILOM start / stop commands.Example: stop /SYS/PM1Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#37New T5 System Firmware featuresSystem Firmware redundancyBoth firmware's are stored in pairs of flash banks.Loading firmware is always done into the unused bank(s). When the load is complete, the system reboots and swaps banks.Support for a firmware load with the host system powered on; BUT, the banks cannot be swapped until the host is powered off.To load into the unused banks:-> cd /SP/firmware/backupimage/-> load To swap the banks:-> set /SP/firmware sysfw_bank_switch=trueMUST be followed by an SP reboot to take effect:-> reset /SPCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#38T5 Platform hardware managementAbility to update Power Supply Firmware (from service)DVFS Power ManagementPower throttling done in real-time by the FPGA based on power consumption, current draw (IWARN), and temperature readings.ILOM out of the loop, other than programming the power thresholds.For PM power load balancing, ILOM set thresholds each second.IFC Fan ControlILOM controls fans speeds using the IFC algorithm.Based on temperature readings across the system (DIMMs, CPUs, etc).Power-on failure fault diagnosisPOK signals are expected to assert when power-on is requested.ILOM can now better diagnose POK assert failure to suspected FRU.Power Glitch fault diagnosisPower monitored for glitches in nanosecond resolution by the FPGA.ILOM creates faults in response to FPGA notification of glitch events.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#39New to ILOM 3.1Simplified Data Model (SDM)Three-Level ModelLevel One: System Summary InfoLevel Two: Subsystem SummaryLevel Three: Logical TopologySubsystems: Cooling, Power, CPUs, Memory,Storage, and NetworkingAlso Blades, DCUs, CMUs, CPU Modules, I/O Modules on some platformsOpen Problems unifies fault management with SDMCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#ILOM 3.2: New Linux distro and compilersM5 and T5 will release with ILOM 3.2Linux version 2.6.27.43, SQUEEZEwas 2.6.16.4, SARGEgcc version 4.4.5was 3.3.6Why:Old distro no longer supportedSecurity and bug fixesPosix threads instead of Linux threadsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#CLI Changes from T3/T4 ILOMDifferent method for disabling componentsBefore: component_state property represented both the current state (disabled by POST or hostconfig) and user-requested state.After: split states to reduce confusioncurrent_config_state = actual state of the resource in the systemdisable_reason = human readable reason of why it's disabledrequested_config_state = user requested stateAs before, must start or reset the host for requested_config_state changes to take effect.requested_config_state can only enable components disabled by requested_config_state.All other disabled reasons are faults, which must be addressed (via fmadm acquit) or the FRU replaced.A fault in one component may cause other components to be disabled. These will be noted with disable_reason of Configuration rules.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#42Other Changes from T3/T4 ILOMBBR (Black Box Recorder) data movednow in /large, a 64MB filesystemBBR now records ILOM dataILOM RAM and filesystems available spaceFor all critical processes: memory usage, #threads, #files open, cpu usageEP (Electronic Prognostics)Solaris fetches EP configuration files from ILOM via SNMPAbility to disable Host console logging-> set /HOST/console logging=disabledDisabling the logging also deletes all stored logs.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#43WEBCLILUMAINSDM BACKENDCAPISSM APILIBHDLHw servicePlatform xmlSDM ArchitectureCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM CLICLI is reorganized./System target(tree) is introducedDifferent components of the system are grouped and organized into sub targets of /SystemAt every level of the tree, the critical properties are shown along with any sub targetsThe applicable cli commands are supported at different levels of the /System tree.All targets and properties under /System are case insensitiveCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM CLI (cont.)Health and health details are two of the common properties shown at every level to indicate the over all health of that sub tree.Open_Problems target shows the detailed descriptions of the faults in the system/SYS and /Storage targets are made legacyContinue to exist but hidden by defaultThe legacy targets can be made visible by enabling /SP/cli/legacy_targets property.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM CLI (Summary level targets)-> show /System -d targets/SystemTargets:Open_Problems (0)ProcessorsMemoryPowerCoolingStorageNetworkingPCI_DevicesFirmwareBIOSCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM CLI (Summary level properties)-> show /System -d propertieshealth = OKhealth_details = -open_problems_count = 0type = Rack Mountmodel = Exadata X2-3part_number = 8124854serial_number = 2229CNL124component_model = SUN FIRE X4170 M3component_part_number = 7013743component_serial_number = 1118CNL013system_identifier = sysidentifiersystem_fw_version = 3.1.0.10primary_operating_system = Not Availablehost_primary_mac_address = 00:21:28:d5:c0:b2ilom_address = 10.153.55.201ilom_mac_address = 00:21:28:D5:C0:B6locator_indicator = Offpower_state = Offactual_power_consumption = 5 wattsaction = (none)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM CLI (Processors Subsystem)-> show /System/Processors/Targets:CPUsProperties:health = OKhealth_details = -architecture = x86 64-bitsummary_description = Two Intel Xeon Processor E5 Seriesinstalled_cpus = 2max_cpus = 2-> show /System/Processors/CPUs/Targets:CPU_0CPU_1

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM CLI (Processors Subsystem)-> show /System/Processors/CPUs/CPU_1Properties:health = OKhealth_details = -part_number = 060Dserial_number = Not Availablelocation = P1 (CPU 1)model = Genuine Intel(R) CPU @ 1.60GHzmax_clock_speed = 1.600 GHZtotal_cores = 8enabled_cores = 8Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM WebDifference from ILOM 3.0:Old components, sensors and indicators pages have been removed and replaced with new subsystem pages.New subsystem pages focus less on raw sensor data, more on status of the components.Old RAID pages are now replaced by the storage subsystem page.Session timeout page has been merged into the web server configuration page.Open problems page brings together all system problems in a single spot, replaces old fault management page.New comprehensive summary page.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#ILOM 3.1 Web redesignNavigation tree on left replaces old tabs on top.Top levels of tree include new summary and subsystem pages.Pages organized by purpose:System InformationRemote ControlPower, Host, System and ILOM AdministrationCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#New Embedded BUI Mini-Help As Of ILOM 3.2.1

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#53Web Summary PageGeneral Information Table: Basic info about Server and SPAction Table: Quick access to common ILOM actions (Power on/off, OSA, JRC, Firmware update)Subsystem status table: Quick summary of the various subsystems.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM StorageNew location for storage information as the previous Storage Viewer UI(/STORAGE/raid in CLI, Storage RAID tab in BUI) is now legacyDifferences from legacy Storage ViewerContains similar information, but simplified and now reflects health of all componentsContains non-RAID Controller and Expander informationCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM Storage Example-> show /System/Storage//System/StorageTargets:DisksControllersVolumesExpandersProperties:health = Not Availablehealth_details = Comprehensive Storage monitoring is not available.Ensure the host is running with the HardwareManagement Pack. For download details go tohttp://www.oracle.com/technetwork/server-storage/servermgmt/downloads/index.htmlinstalled_disks = 1max_disks = 8installed_disk_size = Not Availablelogical_volumes = Not Availabledisk_controllers = Not AvailableCommands:cdshowCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM Storage Example-> show /System/Storage/Disks/Disk_2/System/Storage/Disks/Disk_2Targets:Properties:health = Warninghealth_details = The disk is offline per host request or other reason(disk is not compatible for use in volume). Type 'show/System/Open_Problems' for details.part_number = ST914602SSUN146Gserial_number = 0998SX3L 3NM8SX3Llocation = HDD2 (Disk 2)type = HDDmanufacturer = SEAGATEcapacity = 136 GBdevice_name = /dev/sdcraid_disk = falsewwn = 0x5000c500130310a3Commands:cdshowCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM Storage Example-> show /System/Storage/Controllers/Controller_0//System/Storage/Controllers/Controller_0Targets:Properties:health = OKhealth_details = -serial_number = 500605b001090990type = SASmanufacturer = LSI Logicmodel = SG-XPCIE8SAS-E-ZCommands:cdshowCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM PCI DevicesProvides on-board and add-on PCI Device informationOn-board device description is providedAdd-on information is new to ILOM UIs and includes device description, part number, and PCI IDsThis information is provided to ILOM per slot by BIOS originally for Fan Control. The PCI IDs provided allow for the part number and device description to be knownAdd-on components are based on PCIE slots on rackmounts and PEM, REM, and FEM slots on bladesAdd-ons not supported by a platform are present but the description/PN are Not RecognizedCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM PCI Devices Example-> show /System/PCI_Devices/On-board/Device_0/System/PCI_Devices/On-board/Device_0Targets:Properties:description = NET0 Intel X540 Gigabit Ethernet ControllerCommands:cdshowCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM PCI Devices Example-> show /System/PCI_Devices/Add-on/Device_4//System/PCI_Devices/Add-on/Device_4Targets:Properties:part_number = SGX-SAS6-R-INT-Zdescription = Sun Storage 6 Gb SAS PCIe RAID HBA, Internallocation = PCIE4 (PCIe Slot 4)pci_vendor_id = 0x1000pci_device_id = 0x0079pci_subvendor_id = 0x1000pci_subdevice_id = 0x9263Commands:cdshow-> show /System/PCI_Devices/Add-on/Device_4//System/PCI_Devices/Add-on/Device_4Targets:Properties:part_number = Not Recognizeddescription = Not Recognizedlocation = PCIE4 (PCIe Slot 4)pci_vendor_id = 0x8186pci_device_id = 0x105fpci_subvendor_id = 0x108epci_subdevice_id = 0x115fCommands:cdshowCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM Open ProblemsProvides access to fault information in all levels of SDM UI, system, subsystem, componentOpen Problems UI contains the most in depth information including time, subsystem, location, UUID, description, P/N, S/N, and Knowledge Article URLData retrieved from fault management (similar data to /SP/faultmgmt shell fmadm faulty command) and for storage specific faults from HMPCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM Open Problem Example-> show /System/Open_ProblemsOpen Problems (1)Date/Time Subsystems Component------------------------ ------------------ ------------Fri Jan 20 16:16:48 2012 Memory P0/D8 (CPU 0 DIMM 8)A memory uncorrectable ECC fault on a DIMM has occurred. (Probability: 100UUID: 7df1056b-e208-c9fd-c1bd-e9127d6c05d2, art Number: 001-0003,Serial Number: 00CE021038834C9C59, Reference Document: http://www.sun.com/msg/SPX86-8001-U5)

-> show /System/ health health_details/SystemProperties:health = Service Requiredhealth_details = P0/D8 (CPU 0 DIMM 8) is faulty.Type 'show /System/Open_Problems' for details.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SDM Open Problem Example-> show /System/Memory/ health health_details/System/MemoryProperties:health = Service Requiredhealth_details = P0/D8 (CPU 0 DIMM 8) is faulty. Type 'show/System/Open_Problems' for details.

-> show /System/Memory/DIMMs/DIMM_8/ health health_details/System/Memory/DIMMs/DIMM_8Properties:health = Service Requiredhealth_details = A memory uncorrectable ECC fault on a DIMM hasoccurred. Type 'show /System/Open_Problems' fordetails.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SPARC Virtualization Technologies for The T5Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Oracle Solaris and SPARC VirtualizationBetter Resource Utilization for a More Efficient DatacenterDynamic DomainsOracle VM Server for SPARCM-SeriesT-Series, M5AppAppOracle Solaris ZonesOracle Solaris DW DBDomain ADomain BOLTP DBOLTP DB

AppAppDomain ADomain BDomain CWeb

Oracle Solaris 8 ZoneOracle Solaris 9 ZoneOracle Solaris ZoneOracle Solaris ZoneWebDBAppWebWebWebCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Oracle offers a full portfolio of virtualization solutions to address your needs. SPARC is the leading platform to have the hard-partitioning capability that provides the physical isolation needed to run independent operating systems. Many customers have already used Oracle Solaris Containers for application isolation.

Oracle VM Server for SPARC provides another important feature with OS isolation. This gives you the flexibility to deploy multiple operating systems simultaneously on a single T-Series server with finer granularity for computing resources. For SPARC T-Series processors, the natural level of granularity is an execution thread, not a time-sliced microsecond of execution resources. Each CPU thread can be treated as an independent virtual processor. The scheduler is built into the CPU, without the extra overhead for scheduling in the hypervisor. What you get is a lower- overhead and higher-performance virtualization solution.

Your organizations can couple Oracle Solaris Containers and Oracle VM Server for SPARC with the breakthrough space and energy savings afforded by Oracles SPARC T-Series servers to deliver a more agile, responsive, and low-cost environment.

Virtualization on T5 SystemsHigh degree of virtualizationOVM for SPARC (i.e. Logical Domains) for hypervisor-based virtualizationOracle Solaris Zones for OS virtualizationOracle Enterprise Manager Ops Center provides an administrator-friendly integration of these different virtualization levelsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#21st Century Cloud InfrastructureSPARCOracle VM Server for SPARCOracle Solaris 11Oracle Solaris 10

Solaris 11 ZoneSolaris 11 ZoneSolaris 10 ZoneSolaris Legacy ZoneSolaris Legacy ZoneSolaris 10 Zone

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Virtualization is built on SPARC: OVM for SPARC (LDoms) and Solaris ZonesLegacy Zones were previously called Solaris 8 and Solaris 9 Branded Containers

71Oracle Solaris ZonesSame virtualization technology for all SPARC, x86 systemsSimple; lowest overhead; highest performanceIdeally suited to leverage multithreading hardwareMission-critical deploymentsLargest Sun financial and Telco customersall run Oracle Solaris ZonesIn production on 25+% of installed Oracle Solaris systemsIdeal for a variety of scenariosLightweight test environmentsDynamic environments with resource sharingRapid prototyping test beds on same hardware and OSZones cloning/migration/instant restartBuilt-in Virtualization on Any Oracle Solaris SystemCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Value Prop:Built-inHighly efficientLeverages other Oracle Solaris technologiesZFS, predictive self healing, security, etc.72Built-in VirtualizationOracle Solaris 11 ZonesSecure, light-weight virtualizationScales to 100s of zones/ nodeDelegated administrationZFS datasets, boot environmentsObservability via zonestatSolaris 10 ZonesNFS ServerNetwork stack isolation andresource managementCo-engineered with installation, security, ZFS, networking, IPS, SPARC and x86 hypervisors

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#See

Oracle Solaris Zones continue to be the best environment for deploying applications, and consolidating services. In Oracle Solaris 11, significant work has been done to better integrate this technology in the OS with a much more complete experience, perfect for cloud deployments. With improved observability within non-global zones, the ability to delegate administration to a non-global zone including a complete separate network configuration using virtualized networking, zones are more powerful than ever.Zones also represent the main migration capability with the introduction of Oracle Solaris 10 Zones, allowing administrators to migrate existing Solaris 10 physical systems (and those also running zones) to a similar environment running on top of Oracle Solaris 11.Cloud-Scale NetworkingVirtualize, consolidate network infrastructureIncrease performance and reduce costsSecure IsolationIntegrated functionalityRouting, Firewalling, Load Balancing, Bridging, High Availability

Parallel networking stack. Built to scale.Hardware assisted Network Resource ManagementOptimized for performance at every levelEase of UseAutomatic Networking modeFine grained observabilityVLAN isolation, dynamic VLAN provisioningCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Optimized forSPARC &Oracle SolarisOracle VM Server for SPARCThe Virtualization Platform combining the best of Oracle Solaris and SPARC for Your Enterprise Server Workloads

Optimized forSPARC &Oracle SolarisIsolated OS and applications in each logical (or virtual) domainFirmware-based hypervisorEach logical domain runs in dedicated CPU thread(s)T5 ServerOracle Solaris 10Oracle Solaris 11Database DomainOracle Solaris 10Oracle Solaris 11Database DomainOracle Solaris 10Oracle Solaris 11Database DomainOracle Solaris 10Oracle Solaris 11Database DomainGP DomainGP DomainGP DomainGP Domain

SPARC Hypervisor

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#7575Oracle VM Server for SPARC, previously called Sun Logical Domains, provides highly efficient, enterprise-class virtualization capabilities for Oracles SPARC T-Series servers.

Oracle VM Server for SPARC leverages the built-in hypervisor to subdivide system resources (CPUs, memory, network, and storage) by creating partitions called logical (or virtual) domains. Each logical domain can run an independent operating system.

Oracle VM Server for SPARC provides the flexibility to deploy multiple Oracle Solaris operating systems simultaneously on a single platform.

This is the virtualization solution that fully optimizes Oracle Solaris and SPARC for your enterprise server workloads. Alignment with SPARC designed for ThreadsTraditional VM based on assumption CPUs are scarce, so we must over-commit and time-slice themOverhead for time-slicing different contextsIntercept for privileged operationsLatency servicing every interruptT5/M5 systems are thread-rich - so we can dedicate CPU threads to each domain for native CPU performanceEliminates CPU latency and overheadContext switches in a single clock on cache miss or intervalSome VM systems also over-commit RAM, Causes overhead and requires complex memory managementOk for lightweight, occasional workloads, very bad for enterprise appsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#7676Competing VM-based virtualization technologies bloat their hypervisor with schedulers and workload managers. This is not needed on Oracle SPARC servers because of the multi-core design and the high thread count. Oracles hypervisor is much smaller and more efficient. A smaller hypervisor is also more reliable (fewer lines of code).

Hypervisor SupportHypervisor software/firmware responsible for maintaining separation (eg: visible hardware parts) between domainsUsing extensions built into sun4v CPUResides in the firmware, not the ILOMProvides Logical Domain Channels (LDCs) so domains can communicate with each otherMechanism by which domains can provide services to each otherA protocol lets hypervisor and domains queue and dequeue service request messagesService domains use these channels and owns I/O resources for bridged accessCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#7777Roles of DomainsControl domain Creates and manages other logical domains and services Control domain usually also a service and I/O domainI/O domains own physical I/O bus or devices. May run apps using physical I/O for native performanceService domainsprovide virtual network and disk devices. Typically an I/O domainGuest domain:run applications on virtual I/O devices provided by service domain Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#7878In the current, first, release of Logical Domains, the Control domain is also required to be a service domain, this may change with later updates to the ldoms technology.

With the T2000 systems, the architecture limited the number of service domains to two; can go up to 4 with T4-4 or T5440-4

Domain ComponentsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Hypervisor BasicsWhat is hypervisor?Primary roles:Implements software component of sun4v virtual machine, providing low overhead hardware abstractionEnforces hardware and software resource access restrictions for guest, including inter-LDom communication, to provide isolation and securityPerforms initial triage and correction of hardware errorsSecondary roles:Implements dynamic LDom reconfigurationProvides data for performance statisticsManages hardware elements of some power management featuresCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#80Hypervisor BasicsWhat hypervisor isn't:An Operating SystemHV does not time slice between guests running on strands and has a a fixed memory footprint (no malloc)HV only executes in response to a specific subset of traps. Except where hardware access is involved, traps go directly to the guest for maximum performance. There are separate HV and guest traptables for this reason.A policy makerHV enforces boundaries, but does not define themHV will do as requested, even if it may harm the guest, as long as it does not violate resource access restrictionsThe IO managerDrivers in the guest manage the PCIE fabric and devicesHV enforces access restrictions to IO resourcesCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#81Hypervisor and Logical DomainsOracle VM for SPARCA logical domain is a virtual machine comprised of a discrete logical grouping of resourcesEach Guest runs its own instance of SolarisEach Guest can be created, destroyed, reconfigured, and rebooted independentlyThe hypervisor enforces the partitioning of the server's resources, and the OS and applications running in those partitions (i.e. Guest )The hypervisor allocates a subset of the overall CPU, memory, and I/O resources of a server to a given logical domainUp to 128 guests per hypervisorCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Typical Simple ConfigurationOne control, service, I/O domain: the primary domainServices in the primary domain:A virtual switch (vsw) associated with the primary NICA virtual disk service (vds) exporting vdisk for all guestsA virtual console concentrator (vcc)Other domains are guests with vnets and vdisksserviced by the primary domainGuests consoles are available through the primary domainPrimary console is available through the SPLife cycle of a domain: define it, bind resources to it, start itCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#8383What's new in 3.0 / 3.1SR-IOV and DIO for non-primary Root Domains (3.1)Dynamic SR-IOV, DIO and PCIe busses (3.1)Support for Oracle VM ManagerLive Migration in Elastic ModeDRM in Elastic ModeBoard DR for T5-4 & T5-8 (3.1)Preserve Whole Core Constraint across Live MigrationvNICs on vNet (Zones in LDoms) (3.1)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#84T5 Hardware features relevant to LDomsT5-2 will allow 2 independent IO-Domainsdisks on pci_1 and pci_2network on pci_0 and pci_3T5-4 and T5-8 will allow CPU Board-DRunclear if at release or laterTwo PCIe root complexes per socketmore granular Root DomainsBetter power managementCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#85Example: IO Granularity on T5-2NAME TYPE BUS DOMAIN STATUS---- ---- --- ------ ------pci_1 BUS pci_1 primarypci_0 BUS pci_0 primarypci_3 BUS pci_3 primarypci_2 BUS pci_2 primary/SYS/MB/PCIE5 PCIE pci_1 primary EMP/SYS/MB/PCIE7 PCIE pci_1 primary EMP/SYS/MB/SASHBA1 PCIE pci_1 primary OCC/SYS/MB/PCIE1 PCIE pci_0 primary EMP/SYS/MB/PCIE3 PCIE pci_0 primary EMP/SYS/MB/NET0 PCIE pci_0 primary OCC/SYS/MB/PCIE6 PCIE pci_3 primary EMP/SYS/MB/PCIE8 PCIE pci_3 primary EMP/SYS/MB/NET2 PCIE pci_3 primary OCC/SYS/MB/PCIE2 PCIE pci_2 primary EMP/SYS/MB/PCIE4 PCIE pci_2 primary EMP/SYS/MB/SASHBA0 PCIE pci_2 primary OCC/SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary/SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary/SYS/MB/NET2/IOVNET.PF0 PF pci_3 primary/SYS/MB/NET2/IOVNET.PF1 PF pci_3 primaryCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#86Root Domain1Guest DomainGuest DomainRoot Domain2PCIeswitchPFVFsPCIeswitchPFVFsPF Device driverkernelkernelMultiPathingSolarisI/O stack

VFPF Device driverAppAppkernelVFkernelMultiPathingVFVFAppAppAppAppAppAppVirtualPCIeswitchVirtualPCIeswitchVirtualPCIeswitchVirtualPCIeswitchSolarisI/O stackHypervisorpci@400pci@500pci@500pci@400pci@500pci@400Non-Primary Root domain example configCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#87Resource Management Improvements

Dynamic resource management (DRM) between domainsDynamic CPU movement is based on the priority property of each domain's DRM policy.Ensures that domains running the most important workloads get priority for CPU access over domains with less critical workloadsGives last remaining CPUs to the higher priority domainRemove CPUs from a lower priority domainCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#88Through the enhancements to domain migration and resource management, users of Oracle SPARC T-series servers can benefit from increased application service levels.

Oracle VM Server for SPARC 2.1 delivers:

Live migration: Enables users to migrate an active domain to another host machine while maintaining application services to users. Live migrations are as simple as point and click using Oracle Enterprise Manager Ops Centers console.Secure, encrypted migration included: On-chip cryptographic accelerators deliver secure, wire speed encryption capabilities for live migration without any additional hardware investments.Dynamic Resource Management (DRM) between domains: Ensures that domains running the most important workloads get priority for CPU access over domains with less critical workloads.Increased maximum number of virtual networks per domain: Permits a dramatic increase in external access to domains.Lower-overhead, higher scalability networking for Oracle Solaris 11 initial release: Allows virtual network devices to use shared memory to exchange network packets, enabling improved performance and scalability.Support for Virtual Device Service Validation: Immediately validates the name and path for a specified network device or virtual disk, greatly reducing the risk of incorrectly configured I/O. Integrated Dynamic Reconfiguration (DR) of Cryptographic units and virtual CPUs: Cryptographic units and CPUs are dynamically reconfigured together to simplify operations and ensure consistent performance.Enhanced Management Information Base (MIB): Enables the SNMP MIB to use the latest Logical Domains Manager XML interface, permitting third party management software to access the new features and resource properties. P2V tool enhancements: Bring more flexibility to quickly convert an existing SPARC server running Oracle Solaris 8, 9 or 10 into a virtualized Oracle Solaris image to run on SPARC T-series servers.

Logical Domain Channels (LDC)Logical Domain Channelsprovide low level data services between componentsimplemented in srampoint-to-pointLDM to hvLDM to spsp to Solarisallow passage of large chunks of data more efficiently than mailboxesCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#PCI-SIG Single-Root IOVStandardize way of bypassing the VMM's involvement in data movement by providing independent memory space, interrupts and DMA Steams for each VMBenefitsNative IO PerformanceProvides scalabilityDrawbacksHard to live migrate VM, somewhat similar to Direct assignment

Virtual NICGuest OS0APPAPPAPPVMMPhysical NICVirtual NICGuest OS2APPAPPAPPVirtual NICGuest OS1APPAPPAPPIntel VT-xIntel VT-dCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#90SR-IOVIOV for PCI Express (PCIe) HW An IOV solution that allows direct access to PCI Express devices at Virtual Function (VF) granularity from a Guest DomainStandard for PCIe Fabric with a Single Root-Complex (SR-IOV). Standard for PCIe fabric with multiple Root-Complexes (MR-IOV)FeaturesDirect access to VF registers, interrupt, DMAUsage modelIndividual NIC port belong to different OSesMultiple Guests share SR-IOV devices

VMMVirtual NICGuest OS0APPAPPAPPFn0PFn0VFn0VFn1VFn2System DeviceConfig SpaceVM DeviceConfig SpaceIntel VT-dPhysical NICIntel VT-xCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#91IOV BenefitsPerformanceFully utilize IO device resources such as 10G NIC bandwidthLow latency Cost reductionCapital and Operational Expenditure savings from Power savings, reduced adapter count, less cabling and fewer switch portsBut Migration is disabled once VFs are assigned to a domain.This may change in the future.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#92A High-Level View of VFs

pci_0pci_0pci_0pci_0pci_0PCIe Switch(virtualized)PCIeSwitch PCIe Switch(virtualized)PCIe Switch(virtualized)PCIe Switch(virtualized)I/O Domain 0I/O Domain 1I/O Domain 2I/O Domain 3PrimaryOperatingSystemsOperatingSystemsOperatingSystemsOperatingSystemsOperatingSystemsHypervisorVF0VF1VF2VF3VFsSR-IOV CardCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#93

Secure Live MigrationLive migration available on SPARC systems SPARC M5SPARC T5SPARC T4SPARC T3UltraSPARC T2 PlusUltraSPARC T2On-chip crypto accelerators deliver secure, wire speed encryption for live migrationNo additional hardware requiredEliminates requirement for dedicated networkMore secure, more flexible

VMExternal Shared StorageSPARC T-Series serversOracle VM Server PoolVMSecure Live Migration (SSL)VMVMEliminates Application DowntimeCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#94Live migration: Enables users to migrate an active domain to another host machine while maintaining application services to users. Secure, encrypted migration included: On-chip cryptographic accelerators deliver secure, wire speed encryption capabilities for live migration without any additional hardware investments.

Other products (including VMware) migrate VM data in the clearRequires dedicated networkLeaves sensitive data vulnerable (passwords, account numbers, etc.)

Cross CPU Migration - ArchitectureAllows migration of domains across sun4v architecture platformsSupports migration among platformsWill be extended to support new platforms as they are introducednew platforms might not be migration-compatible with all previous platformsAllows migration among same CPU architecture with different system clocks frequenciesDependent on guest domain having Solaris 11Solaris introduces a generic sun4v CPU module, simulated 1GHz system clock if HW not available, other changes.LDoms Manager introduces domain cpu-arch property95Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#95Cross CPU Migration - Solaris Supported in guests runningSolaris 11 U1Solaris 10 U11Introduces new generic CPU module: sun4v-cpuDomain service extension to identify CPU module capabilitiesCPU Module has a major/minor version number, used by domain manager to determine capabilities of the guestSimulates 1GHz system clock if neededKernel routines for read tick/stick modified to emulate clock rate: emulate 1GHz in generic mode; emulate boot frequency after migration in native mode to system with different clock frequency96Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#96Cross CPU Migration Generic DomainsLDom Manager/Firmware/Solaris must be of sufficient revision to support Cross CPU MigrationFirmware must support LDom Live Migration on both source and target domainsGuest domain must be Solaris 11 FCS or newerMigration is for the most part unchangedAt the start of the migration, domain capabilities and generic CPU module version are retrieved and sent to the targetCheck on target ensures that the target processor supports the generic CPU module version97Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#97Live Migration ImprovementsPreviously, domains lost the whole-core constraint when migratedNo more hard partitioning => violation of license capping rulesWith 3.0, whole-core constraint is preservedAllows migration of hard partitioning domainsMemory-DR after migration is now enabledrequires Solaris 11.1 or Solaris 10 update 11Live Migration in Elastic ModeLive Migration without password required for OVM ManagerNeeds to be enabled per systemCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#98Solaris Support

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Solaris Support for T5 PlatformSolaris Plan of Record for T5 RR:S11.1 + SRU3 (pre-installed)S10U11LDoms 3.0Getting Solaris release information (what version is installed) :Use pkg list kernel & pkg list entireFor RR, will qualify and support S10U9 and S10U10 + patch bundle in a guest domainCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Platform Solaris Overview of changesT5 performance counter additions/modifications"busstat -l" displays the number of drams on the system. The "busstat -e dram" displays list of events for dram countersThere are 4 cpu performance counters to count the events listed in "cpustat -h" output.3072 NCPU sun4v supportEnhanced sun4v kernel to support MPO for large configCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Platform Solaris Overview of changesEnhanced sun4v kernel to support suspending & resuming entire running OS instance (including I/O) to support DRMade sun4v kernel resilient to allow/continue booting if it couldnt start a cpu and use remaining cpusCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Power Management

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Power Management Features

FeatureT4T5M5CommentsDynamic Voltage & Frequency Scaling(DVFS)No New to SPARC, already exists on x86Cycle Skipping T4 whole socket granularityT5/M5 sub socket granularityCoherency Link ScalingNo* NoT5-2 onlyPower SuppliesGold+A261AA254T5 PS (A261): Goal PlatinumM5/M6 (A254): 3 Phase goal similar to platinumIFS (Intelligent Fan Control) (Technically not CPU feature)Existing M Series (M3000-M9000) have no active power management featuresCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Power management is critical to the Cloud initiatives by Oracle.DVFS Power throttling done in real-time on power consumption, current draw (IWARN), and temperature readings.

Coherency Link Scaling for the T5-2 While 4 links are active, 2 can be turned off to save up to 16W

104Coherency Link ScalingT5-2 has 4 Coherency Links connecting the socketsIn Elastic Mode only, up to 2 of these can be turned offSavings up to 16WNotes:Other T5 servers have do not have sufficient links to turn offCannot be used in Performance ModeExclusive to T5-2 ONLYCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Power Management Interfaces

ssh CLIHTTP ILOM

11.1 poweradm10 & 11: pwconfig & /etc/power.confSolaris

Rich Choice of Management Options Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Software Component InteractionPowerAwareDispatcherGuest (Solaris)Control Domain/LDoms ManagerSPpstatesHVCoh-link scaling,Coordinate PADCPUDIMMBoBCPUPMMCUhostconfigmempmcstatesPowerCapperSystem & HW domain capSystem domain policyPM PolicyPM policyAffinityEnginePM policyFPGAchannelPM capperPRIDVFS cycle skipPower capDVFS cycle skipPlatform MDBoB LinkCoherencyLinkInitializeInitializeInitializeInitializeDVFS cycle skipCoherency LinkScalingPPFECopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#107PRI = Physical Resource Inventory

ILOM 3.2 Power Policies3 Policies:Disabled: all components run at full speed (old performance policy)Performance (default): unused components power managedunused components are power managedpower savings features with insignificant performance impact are enabledElastic: unused or idle components power managedCPUs, cores, memory, (coherency links - T5-2 only)Prior versions of ILOM < 3.2 had 2 PoliciesPerformance (equivalent of new disabled policy), and Elastic

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#System (PDom) Power Management PoliciesThe PM policy is managed in ILOM under the /SP/powermgmt target. There are several ways to view or change the policy (browser, command line, Ops Center).Ops Center can only set to performance or elasticILOM Command Line (login as root)-> show /SP/powermgmt policy-> set /SP/powermgmt policy=elastic-> set /SP/powermgmt policy=performance-> set /SP/powermgmt policy=disabledCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#System (PDom) Power CappingShow the power cap settings:-> show /SP/powermgmt/budgetShowthe current power consumption:-> show /SP/powermgmt actual_powerConfigure pending power limit in watts (replacing 400 with a value that is appropriate for your environment), use:-> set /SP/powermgmt/budget pendingpowerlimit=400To apply the pending values, use:-> set /SP/powermgmt/budget commitpending=trueTo enable the configured power limit, use :-> set /SP/powermgmt/budget activation_state=enabled

ILOM 3.2 CLICopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#System (PDom) Power Capping in T5 and M5Soft Cap:limits average power consumptionHard Cap:T5: stays within blade system or PDU power constraintsM5: physical domain boot and board add prevented if hard cap would be exceeded

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#you can set a hardcap via the CLI also, by setting pendingtimelimit=0

111Integrated System (PDom) & OS PM PolicySystem policy applied to all guest OSs/LDoms (default)Administrator can override on local S11u1 OS via Solaris poweradm commandImpacts that LDom/Guest exclusivelyImpacts shared resources such as memoryAdministrator can only set System Policy via ILOM Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Solaris 11.1 Power ManagementPower Aware Dispatcher (PAD) for SPARCAlready available in Solaris 10 for x86More adaptive scheduling, better performance and efficiencyApplies DVFS to idle processorsApplies cycle skipping to idle coresEnabled by defaultPerformance (default) or Elastic policies enable Power Aware Dispatcher.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#To avoid ambiguity by Applies DVFS on idle Processors Processors refers to the entire CPU taking up the socket

113Solaris 11.1 Power Managementpoweradm(1M) replaces pmconfig and /etc/power.confStore configuration data in SMF, instead of configuration filesSecurity implemented using RBACnew properties:time-to-full-capacity, time-to-minimum-responsiveness, suspend-enableadministrative-authoritysmf: Solaris instance has controlplatform (default): the platform has control (eg LDOMs)none: power management is disabled

poweradm commandCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Power Management ObservabilityILOMPower consumption history and graphsBreakdown by physical domain (M5), component typeLDomsPer guest CPU power consumption based on CPU utilizationPer guest memory power consumption based on memory allocationSolarisPowertop pstate and cstate residencyCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Hypervisor Runtime ExecutionHypervisor arbitrates DVFS and cycle skip requests:Power Aware Dispatcher (PAD) guests may request different virtual pstatesLDoms Manager monitors CPU utilization of non-PAD guests and requests DVFS and cycle skip adjustments HV resolves all requests to create the HW DVFS pstates and cycleskip ratiosHypervisor monitors and adjusts per-resource power levels:Coherency link scaling based on coherency link traffic

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#116

Logical Domains ObservabilityCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Ops Center Power Management FeaturesManagementSet policy 1:1Set policy on group of serversMonitoringCurrent consumptionHistoryGraphingAverage by group of servers

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Ops Center Responsible Energy Policy$KwH, WATT, CPUIn/Out Pull, Temperature See Real Cost in Real TimeRelationship between utilization and energy consumptionEnforce Energy Policies Across Servers

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

Ops Center Rack Level Energy ReportsReported from the PDU, custom grouping, and server levelsTop and Bottom ConsumersTurn off Groups of Servers

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Power Management AdvancesHardware saves power below 100% utilization with:Chip wide DVFSPer core pair cycle skippingSerDes power scalingDIMM off-lining w/ Dynamic ReconfigurationDRAM PPSE and PPFE supportPCI Express Power ManagementClock GatingWhen peak performance is demandedPower Management Controller achieves maximum frequency within customer imposed power and thermal limits

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Power vs Frequency with DVFS

Power Management Controller: Elastic SavingsHardware saves power below 100% utilizationChip wide DVFSPer core pair cycle skippingSoftware monitors frequency needs of all coresPuts chip at DVFS point satisfying all cores requirementsPuts core pairs at lowest cycle skip ratio satisfying 2 cores in the pairf(x)=x2.82Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

Coherency Link Power SavingsLink scaling (4,3,2,1 dynamically as needed) Hardware monitors link utilizationSoftware sets entry exit policy (thresholds and dwell times) 4 linksM5/T5M5/T51 linkM5/T5M5/T53 linksM5/T5M5/T52 linksM5/T5M5/T52 linksM5/T5M5/T54 linksM5/T5M5/T525WSavingsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Peak Performance Thermal Management4 thermal diodes per chipcentered in core quads If any T > high-water mark Drop Freq, V If all T < low-water mark Raise Freq, V VDDIWARNalarmsVIDthrottle /resumePLLT5 CPUVIDT5MTemp SensorPMCVRMCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#124

Peak Performance Current ManagementDrop F,V if any current > high-water mark Raise F,V if any current < low-water markControls currents for CPU VDD plus motherboard and DIMMsIWARNVRMVIDthrottle /resumePLLT5 CPUVIDT5MPMCCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#DVFS FunctionalityM5/T5 CPU supports operation at multiple voltage/frequency pairs called pstatesHigh performance (frequency) states require high voltage => higher power statesDVFS allows seamless dynamic switching across pstatesDVFS engine responds to throttle/resume pin toggled by FPGA in response to over-temp/over-current scenarios12V CurrentSensor(High Water)12V CurrentSensor(Low Water)M5/T5 ChipThrottleResumeTemp.ThermalSensorsThermal/PowerContro FPGACurrent LowWater ExceededCurrent HighWater ExceededThermal DiodesCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#DVFSDynamic Voltage Frequency ScalingNot all chips are running at the same voltage/frequencyUp to 32 P-states defined in efuse per chip

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#127DVFST5/M5 supports HW enabled mode (vs disabled)SW programs T5 P-state tables from Efuse and sets limitsFPGA pulses Throttle pin in response to thermal sensor alarmFPGA pulses Resume pin in response to thermal sensor noteThrottle/Resume inform the PMC when to migrate P-states (max delta 200MHz, 6.25mV)Throttle instructs Power Mgmt Controller (PMC) to increase P-state by 1 to reduce power useResume instructs PMC to decrease P-state to lowest allowedT5 transitions up/down P-state table under FPGA controlCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#128EfuseEfuse per chip data set in fabUsed to control what components may be enabled in a processor node (L3 Banks, CPU cores, Serial number)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#129Open BootCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#OpenBoot - Introduction

OpenBootTM is Oracles trademark for Boot Firmware based on the open standard IEEE-1275 for Open FirmwareResident in System Flash (along with Host-Config, Hypervisor, and POST).System independent initialization and boot codeConsumes Guest Machine descriptor which defines the HW configuration for the guestInitialize IO devices and option cards Builds HW configuration in a device tree format for OS clientsBoot OS from disk or networkProvide boot time services to OSCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#131OpenBoot Introduction (continued)

OpenBoot is started by Hypervisor as guest, one instance per guestOpenBoot services retired during OS boot, after control is transferred to the OSOpenBoot binary name openboot.binOpenBoot Component version 4.35.xOpenBoot Binary is common across all M5/T5 platforms, released as part of SysFW packages (SysFW packages are platform specific) Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#132GuestDomainsSun4vAPIOpenBootOpenBootSolaris 10- Kernel/Drivers- FMA AgentSolaris 11- Kernel/Drivers- FMA AgentUser AppsSunVTSUser AppsSunVTSHypervisorPOSTHostConfigSPARC T5 CPUMemoryIOHost HardwareSP HardwareFPGASP CPUGuestManagerILOM / LinuxHost-ConfigHypervisorOpenBootPOSTLinuxILOMGuest ManagerSPARC T5 Software StackCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#133Platform ManagementCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Platform ManagementSNMPOracle Enterprise Manager Ops Center Sun Cluster 3.2 and Sun Cluster 4.0 are supportedCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#135SNMP RequirementsThe SNMP Agent, based on open source Net-SNMP, will run on the SP and export all platform/chassis information relevant for monitoring at the system, component, and domain levelsThe Agent will export read-only MIB information and SNMP traps/notifications from MIBs, including the Platform MIB and Sun Fault Management MIB, to all interested third party managersThe Agent will run SNMPv1, SNMPv2, and SNMPv3. SNMPv3 can be utilized to ensure secure SNMP communication through the authentication, privacy, and access control mechanisms USM and VACMCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#136SNMP Requirements (cont.)By default, the SNMP Agent will not be enabledBy default, the SNMP Agent will have no configuration. When enabled, it run on port 161 with no version support until the configuration is updatedThe Agent will only be enabled on the active SP in a dual SP configurationAll configuration will be handled by specific CLIs to ensure proper SP authorization and auditing. These CLIs will ensure configuration information persistence to support fail-overCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#137SNMP MonitoringComprehensive SNMP Support (V1, V2c, V3)Out of Band via ILOMStandard MIBs RFC1213-MIB SNMP-FRAMEWORK-MIB SNMP-MPD-MIB SNMP-Control-MIB ENTITY-MIB SNMP-USER-BASED-SMMIBOracle MIBs SUN-HW-TRAP-MIB SUN-SUN-HW-CTRL-MIB SUN-ILOM-CONTROL-MIB SUN-PLATFORM-MIB SUN-HW-MONITORING-MIBCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#MIBs for the T5The SP supports 2 MIBs for SNMP:SP-MIB (ILOM extension MIB) - This is used to get information on the status and configuration of the platform. If there is a fault, it send a trap with the basic fault information.FM-MIB (Fault Management MIB) - This is used only when there is a fault. It send the fault trap, but includes all the same detailed information as the FMA MIB in a Solaris domain. The information has the information need by the service technician when placing a service call. Also useful if the domain crashed due to a part failure.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#139MIBs for the T5 (cont.)There are two methods of FMA reporting on the SP: via SNMPthrough the internal network to the affected domain. To have the SP report all platform faults via SNMP using FMA descriptors, you should enable SNMP on the SP.When the command "setsnmp enable" is run, both MIBs are enabled. For SNMP fault reporting, here are the choices:setsnmp enable SP_MIB (Just send SP traps)setsnmp enable FM_MIB (Just send FMA traps)setsnmp enable (Send both)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#140Oracle MIBsSUN-HW-TRAP-MIB Describes hardware related notifications/trapsSUN-SUN-HW-CTRL-MIB Provides platform control via ILOMSUN-ILOM-CONTROL-MIB Controls ILOM devicesSUN-PLATFORM-MIB Oracle specific extension to entity MIBSUN-HW-MONITORING-MIB Provides inventory, status, version, power consumptionCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

Introducing: Oracle Enterprise Manager 11g Ops CenterIntegrated Infrastructure Management +Integrated Application-to-Disk Management+Integrated Lifecycle Management+Integrated Systems Management & Support

Industrys First Converged Hardware Management Solution

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Now available with every Premier support contract, Ops Center is the ideal choice for managing the SPARC hardware, virtualization, and Solaris environments.

142 Manage Your Infrastructure in One Place

Infiniband & Ethernet Fabrics

EngineeredSystems

Storage SystemsOperating SystemsEnterprise ServersVirtualizationExadataExalogicSolaris Cluster

for SPARCContainers

Dynamic Domains,OVM for SPARCCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Required for Sun environments, lower hardware TCOTalk about architectural freedom, allowing you to use the best hardware and manage them all in one place enabled by Ops Centers ability to abstract away the intricacies of the hardware environment. 143Key Features

AutomationActive dependency rulesJob schedulingJob simulationRollback and recoveryDISCOVERInventoryBare-metal discoveryVM auto discoveryAdvanced permission modelTeam sharingPROVISIONFirmwareSolaris and LinuxGolden imagesLDom hypervisorProvision OS in Zones/LDomsUPDATESolaris, Linux WindowsBaseline reportingMatching mirrorIntelligent knowledge basePatch an OS in a VM, Zone, LDomMONITOR/MANAGEHardware and OSResource optimizationReportingAudit logHistorical monitoringCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#144So what are some of the life-cycle management capabilities that Ops Center offers?Ops Center can Discover bare metal and virtual systems automatically and keep track with inventory tools, as well as provide advanced permission models and team sharing.Ops Center can Provision firmware and operating systems (Solaris and Linux) to bare metal systems based on stored golden images, and can provision operating systems to virtual machines (Zones and Ldoms).Ops Center leverages its advanced intelligent knowledge base and dependency engine to update Solaris, Linux, and Windows environments with accurate and timely patches.Ops Center can monitor hardware, OS, and resource utilization and track historical trends, and also provides extensive out of the box reporting and audit capabilities.All of these capabilities are combined with rich automation to help speed processes and reduce errors, and reduce risk in carrying out tasks with capabilities such as job simulation and rollback capability.

Speed VM deployments.Increase productivity.Advanced Virtualization ManagementCentral interface for VM lifecycle managementSolaris Zones, Oracle VM for SPARC, Dynamic DomainsMonitor VM or system-level utilizationReconfigure VMs dynamicallyCreate resource poolsMigrate VMs across serversCentralized VM Lifecycle Management

Dynamic DomainsM-Series

T-SeriesSPARC & X86HYPERVISOR

Oracle VM for SPARCZones

M5-32Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#145Ops Center provides the complete management environment for each of the layers from OS and below.

Ops Center LoginCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5-8 - Summary

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

T5-8 - HardwareCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SPARC RoadmapCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

Oracle SPARC Processor RoadmapSolaris 11 UpdateSolaris 10Solaris 11 UpdateSolaris 10 UpdateSolaris 11 UpdateSolaris 10Solaris 11 UpdateSolaris 10Solaris 11Solaris 10 U10T4+1x Throughput +5x Thread Strength

In TestM-Series+6x Throughput+1.5x Thread StrengthIn the LabM-Series+2x Throughput>1x Thread StrengthM-Series+2x Throughput+1.5x Thread Strength In TestT-Series+2.5x Throughput+1.2x Thread Strength201120122013201420152016DeliveredT-Series+2x Throughput+1.5x Thread Strength Oracle Application AcceleratorsDatabase QueryCompressionEncryptionCluster InterconnectSoftware QualityCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Oracle is the only vendor that makes its processor and OS roadmaps publically accessible.This shows how we aggressively will double performance every 2 years, or less.This allows customer to upgrade their servers without having to wait 3 years or more for the next generation to be released.This also means secondary technology, like DDR3 memory or Gen 3 PCIe, are also upgraded in this accelerated schedule.Instead of updated being released every 3-4 months, updates now come once per year. This gives the customer more soak time in evaluating and integrating an update release.

150Application AcceleratorsDatabase queryCompressionEncryptionCluster InterconnectApplication Data Protection

Increased PerformanceHigher core frequencyMultiple pipelines per coreIncreased core counts per chipLarger cachesMore memory bandwidthPerformance Reliability Security In-memory Database Big DataSPARC Future Directions2x Application Performance Improvement Every 2 YearsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Possible future features of the S4 core and beyond.

151SummaryIn simple terms, moving a binary from T4 to T5 gives at least double or more performance. 2.7x Memory bandwidth, 2x I/O bandwidth of T4.2.4x Throughput over T4, for 128 threads.3.6 Ghz core, Inherit all the advancement of T4 (OoO core, crypto, L2 cache per core...)New directory based protocol for scaling (2/4/8 socket)Significant advancements in power management mean that power consumption will scale well with loadIdle systems will consume a small fraction of peak powerEnterprise-class RAS features.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#153Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#154