Avinash Shukla, Manas Kumar Nath, and Sandeep Kumar

Customer Support Engineers

May 12, 2015

Cisco Unified Computing System (UCS) Mini Architecture, Troubleshooting and Upgrade

Cisco Support Community

Expert Series Webcast

Upcoming Expert Series Webcast

June 23, 2015

To help you more effectively deal with the inevitable troubleshooting scenarios you’ll encounter, this webcast equips you with a 7-step troubleshooting model that allows you to confidently approach and deal with any troubleshooting situation. You’ll also be equipped with efficiency tips for using Cisco IOS “show” commands, and the webcast concludes with a group exercise where you’ll help troubleshoot an issue being experienced on live gear.

Troubleshooting with Speed and

Confidence for Network Engineers

https://supportforums.cisco.com/event/12486661/expert-webcast-troubleshooting-speed-and-confidence-network-

engineers

Kevin Wallace

ACS 5 Installation, configuration, and

troubleshooting, configuration and troubleshooting

with Cisco Expert, Javier Henderson

Now through May 22nd

Ask the Expert Events – Active

Join the discussion for these Ask The Expert Events:

http://bit.ly/events-webinar

High CPU on IOS. Learn and ask questions about

high CPU condition that you might be facing your

environment and troubleshooting the same with the

tools and techniques available within the platform

with Cisco expert Vinit Jain

https://supportforums.cisco.com/expert-corner/top-contributors

Participate in Live

Interactive

Technical Events

and much more

http://bit.ly/1jlI93B

Become an Event Top Contributor

Rate Content Now your ratings on documents, videos, and blogs count give points to the authors!!!

So, when you contribute and receive ratings you now get the points in your profile.

Help us to recognize the quality content in the community and make your searches easier. Rate content in the community.

https://supportforums.cisco.com/blog/154746

Encourage and acknowledge people who generously share

their time and expertise

Cisco Support Community Expert Series Webcast

• Today’s first featured expert is Manas Nath, a customer support engineer from the Server Virtualization team at the Cisco Technical Assistance Center in Bangalore, India.

• Ask your questions now in the Q&A window

Manas Nath Customer Support Engineer

Cisco TAC, Server Virtualization

Tuesday, May 12, 2015

Cisco UCS Mini Architecture,

Troubleshooting and Upgrade

Cisco Support Community Expert Series Webcast

• Today’s second featured expert is Sandeep Kumar, a customer support engineer from the Server Virtualization team at the Cisco Technical Assistance Center in Bangalore, India.

• Ask your questions now in the Q&A window

Sandeep Kumar Customer Support Engineer

Cisco TAC, Server Virtualization

Tuesday, May 12, 2015

Cisco UCS Mini Architecture,

Troubleshooting and Upgrade

Expert Series Webcast: Cisco Unified Computing System (UCS) Mini Architecture, Troubleshooting and Upgrade

Technical Expert, Question Managers

Avinash Shukla

Now through May 22nd, 2015

Ask the Expert Event following the Webcast

Join the discussion for these Ask The Expert Events:

http://bit.ly/events-webinar

Avinash Shukla, Manas Kumar Nath, and Sandeep

Kumar

will be continuing the discussion in an Ask the Expert

event. So if you have more questions, please visit the

Knowledge Center on the Cisco Support Community

https://supportforums.cisco.com/discussion/12500391/ask -expert-cisco-unified-computing-system-ucs-mini-architecture-troubleshooting

If you would like a copy of the presentation slides, click the PDF file link in the chat box on the right or go to:

https://supportforums.cisco.com/document/12500371/expert-webcast-slides-cisco-ucs-mini-architecture-troubleshooting-and-upgrade

Thank You For Joining Us Today!

Submit Your Questions Now! Use the Q & A panel to submit your questions

and the panel of experts will respond.

Please take a moment to

complete the survey at

the end of the webcast

Avinash Shukla, Manas Kumar Nath, and Sandeep Kumar

Customer Support Engineers

May 12, 2015

Cisco Support Community Expert Series Webcast

Cisco Unified Computing System (UCS) Mini Architecture, Troubleshooting and Upgrade

Polling Question 1

Have you performed a Unified Computing System Upgrade?

1. I have done using auto install.

2. I have performed using manual upgrade method.

3. I have never performed a UCS upgrade

4. I want to perform UCS upgrade myself.

• Platform Overview & Software Architecture

• System Functionality

• Feature Specific Details (Licensing, Scalability Ports, USB, Firmware)

• Licensing

• UCS Mini Firmware & Upgrade

• Upgrade Demo

• FC Direct Connect & BFS

• Troubleshooting

• Backup and Restore

• Tips and Tricks

Agenda

Platform Overview & Software Architecture

Interconnect A

IO Modules

Servers

IO Modules

Classic UCS Architecture

Interconnect B

UCS Mini Platform Components

New Chassis UCS-5108-AC2 & UCS-5108-DC2

In-Chassis Fabric Interconnects UCS-FI-M-6324 (With Version 3.0)

Rack Servers C220/240 M3 SB + UCSC-PCIE-CSC-02 (Support for direct attach only; can be connected to scalability port via octopus cable)

Appliance

• In-Chassis Fabric Interconnect – UCS-FI-M-6324.

• Integrated FI and IOM

• 4 x 10G SFP+ (Unified Ports)

• 1 x 40G QSFP+ in 4x10G breakout (Eth/FCoE only).

• 16 x 10G Backplane KR links connected to blades.

• 1x1G Backplane Cluster Link

• 1x10G Backplane Peer Link for data

• USB Port

• 1G Management Port

• Console Port

UCS-Mini Fabric Interconnect

Management

Network port

USB

4 x SFP+

1 x QSFP +

Console Port

Supported Port Roles

Unified Ports - 4 x SFP+ Supported Roles:

Eth/FC Uplink, Server, Appliance, FCoE uplink & FCoE storage

Scalability Port: 1 x QSFP+ (Eth/FCoE only) in 4x10G Breakout at FCS Supported Roles:

Server, Appliance, FCoE Storage (i.e. only south-facing roles)

Management Use Cases

SMB / Branch Office

Cisco ISR Branch Office Router

UCS-Mini

Local Admin

SMB / Branch Office

Cisco ISR Branch Office Router

UCS-Mini

Remote Admin

HQ / Main Office Main

Office Router

Internet (VPN) / WAN

Mini-UCS

SMB / Branch Office

Cisco ISR Branch Office Router

UCS-Mini

Remote Admin

HQ / Main Office

Main Office Router

Internet (VPN) / WAN

UCS-Central (VM)

Autonomous (Local Admin) Autonomous (Remote Admin)

Centralized Management

• Baseline: All features supported in (UCS equivalent) 2.2.1 are supported in 3.0 with some exceptions.

• Unsupported features:

• Private VLANs

• Port Security

• New features:

• Dual Line PSU

• USB support

• QSFP+ (Scalability Port) Breakout Support

• Licensing Support for QSFP+

• HTML5 GUI (Java GUI will continue to be supported)

Supported Features

Software Components

SPRINGFIELD

MEZZ_CARD

PROCESSING_NODE

IO_MODULE IO_MODULE

SPRINGFIELD

BA

SAM

CMC

IBMC

BIOS

Palo/

Menlo

Host

Agent

DCOS

Utilities

CMC

DCOS

UCSM

Switch/N

XOS

Switch/N

XOS

FI-IOM FI-IOM

Adaptor

Firmware

FI-IOM FI-IOM

Fabric-Interconnect UCS-FI-M-6324

Blade

UCSM + Switch Software (NXOS) + CMC(Chassis Management Controller) are co-located on FI

UCSM Central 1.2 supports 100 UCS-Mini domains.

UCS-Central

• 3 Racks directly connected to the primary chassis via Scalability Port

Equipment – Main Topology View

FI-IOM Management Model

• UCS-FI-M-6324 is both an FI and an IO Module • IOM – Blade Connectivity & Chassis Mgmt

Functionality

• FI – Switching Functionality

• Management Model captures both aspects

HIF PC Chassis Backplane Ports, a.k.a Host Interface (HIF) Ports

Front Panel Ports including Scalability Port

FI-IOM Management Model – Port Mapping

NXOS Mapping: Blade 1: Eth1/6-7 Blade 2: Eth1/8-9

.. Blade 8: Eth1/20-21

NXOS Mapping:

Eth1/1-4 Scalability Port: Br-Eth1/5/1-4

NXOS Mapping: PC consists of one or more HIF ports

Internal Ports: eth 1/22 eth 1/23 eth 1/24

FI-IOM Management Model – Thermal Stats

Two new sensors added for UCS-Mini

FI-IOM Management Model – System/Port Stats

• Primary chassis discovery happens automatically upon system reboot. Chassis Discovery policy is ignored.

FI-IOM – Chassis Discovery

Acknowledge Chassis on primary chassis will retrigger chassis discovery.

FI-IOM Hybrid Display

Discovered Backplane Ports

• Background:

• Primary Heartbeat (HB) & Election Messages run over peer link (bond0).

• Secondary Heartbeat runs over chassis SEEPROM when primary HB fails.

• SEEPROM also maintains cluster, chassis, fabric & DB version info.

• Functionally same as classic UCS with the exception of following differences:

• Peer link (bond0) consists of a single 1G link (eth2) on chassis backplane.

• Only Primary Chassis SEEPROM is used for secondary heartbeat. (Racks will not be used for secondary heartbeat.)

High Availability (HA)

HA – Troubleshooting Tip

Upon reboot, this error (-36) appears transiently and should clear after chassis discovery is complete.

If the issue persists, check chassis discovery FSM. ‘Acknowledge’ Chassis may resolve

the issue.

In progress case

success case

• FI-IOM can be hot inserted or removed.

• Identical to classic UCS, during first setup, user must choose either standalone or cluster configuration.

• To fully setup a cluster configuration, user must setup both FI-IOMs. If only a single FI-IOM is setup, it is an incomplete configuration and UCSM will not be accessible.

• Once fully setup, an FI-IOM can be pulled out. This would cause the existing FI-IOM to become primary if it was secondary.

• Note: This is meant to be only for transient condition till removed FI-IOM is replaced.

Online Insertion and Removal (OIR)

OIR – Valid Operations A B

SWAP SWAP SWAP

Cluster Instance IDs match in each of these cases.

OIR – Invalid Operations

Ways to Recover (same as

classic UCS):

• Undo FI-IOM swap

• Erase config on one side and re-cluster

SWAP SWAP

OIR – Invalid Configurations - Standalone A-Standalone B-Standalone A-Cluster B-Standalone

• Standalone + Standalone case – Chassis Discovery will fail.

• Cluster + Standalone case – Chassis Discovery will fail, cluster

mismatch reported on the cluster setup. Ways to Recover (same as classic UCS):

• Enable cluster

• Erase configuration and re-cluster

• Remove one FI-IOM

• UCS-Mini [UCS-FI-M-6324] Supports

• 4 10G SFP Ports (Unified Ports)

• 1 40G Scalability Port

• Scalability Port :

• Octopus-Cable can be used to breakout the 40G Port into 4 x 10G QSFP Ports

Scalability Ports

Equipment-View

• During Chassis/FI-M Discovery,

Ports are discovered.

• 4 SFP Ports are numbered

1/1,1/2,1/3,1/4

• Breakout Scalability Ports are

number 1/5/1,1/5/2,1/5/3,1/5/4

Supported Roles

Scalability Ports Server Port

FCoE Storage Port

Appliance Port

SFP Unified Ports

Server Port

Uplink Port Fcoe Uplink Port

FCoE Storage Port

Appliance Port

Scalability Port Display

Scalability Port Inventory : CLI

Mini-B# scope fabric-interconnect b

Mini-B# show port

Scalability Ports 1/5/1,1/5/2,1/5/3,1/5/4

• 4 SFP 10G Unified Ports

• Can be configured as FC or Ethernet Ports

• Constraints :

• All FC ports must be contiguous.

• All Ethernet ports must be contiguous.

• FC Ports must come before the Ethernet ports in

• the order [1..to..4]

Unified Port Configuration

1 2 3 4

1 2 3 4

1 2 3 4

1 2 3 4

1 2 3 4

Ethernet

FC This behavior is different from UCS where Ethernet Ports come before FC Ports.

Unified Ports Configuration

Unified Port Configuration [ CLI ]

• FC Ports must be configured so that they are before Ethernet

Ports.

Error

Success

• USB can be hot inserted & removed. USB drive is auto-mounted at /mnt/usbslot1.

• USB is supported only for firmware downloads.

• Boot from USB is not supported.

• Export/Import to/from USB is not supported.

• CLI supports the ability to browse USB while GUI does not.

• Only Cisco USB key is officially supported.

USB Support

• It can take up to 1 min to detect insertion/ removal of the USB key.

• Download will fail if user attempts to download immediately after removing the USB key.

USB Support - GUI

When USB is inserted

• Local-mgmt supports all file operations including browsing on usbdrive:/

• Can download image from either side usbA or usbB.

USB Support - CLI

When USB is

not inserted.

Licensing

• Uses existing UCS License Mechanism

• Two Applicable Licenses

• ETH_PORT_ACTIVATION_PKG

• SCAL_PORT_ACTIVATION_PKG

• Includes a 120 Day Grace Period

• Upon Grace Period completion license is expired.

UCS-Mini License Scheme

• Default License (Factory-Installed)

• [ ETH_PORT_ACTIVATION_PKG]

• Cisco UCS 6324, 4 10G Unified Ports, Counted License.

• These can be used for Uplink, Server, Storage Appliance etc.

Default License

• Scalability Port : Additional License

• [ SCAL_ETH_PORT_ACTIVATION_PKG]

• Cisco UCS 6324, 1 40G Eth Port, Counted License.

• Break-out ports can be used for Server (Rack Server), Storage Appliance role

Scalability Port License

• UCS-Mini does support Scalability Port in 40G mode.

• It operates in breakout mode and provides 4x10G Breakout Ports

• SCAL_ETH_PORT_ACTIVATION_PKG gives 1 absolute port count.

• If any of the four breakout ports is configured and enabled, “1” license is used. If all the four ports are un-configured /disabled “1” license is released.

Scalability Port License Handling

Port 1/5/1 configured as Server Port

Scalability Port License : Used

• Admin License Management [SCAL_ETH_PORT_ACTIVATION_PKG]

Scalability Port License : Used

UCS Mini Firmware & Upgrade

• UCS-Mini has 5 images bundles:

• INFRA: ucs-mini-k9-bundle-infra.<version>.A.bin – new

• ucs-6324-k9-kickstart.<version>.bin

• ucs-6324-k9-system.<version>.bin

• ucs-manager-k9.<version>.bin

• BLADES: ucs-k9-bundle-b-series.<version>.B.bin – same as classic UCS

• RACKS: ucs-k9-bundle-c-series.<version>.C.bin – same as classic UCS

• CATALOG: ucs-catalog.<version>.T.bin

• UCS-Central: Release 1.2

• Version 3.0 is not available for classic UCS Infra bundle.

Firmware Management

• UCSM version 3.0 is not supported on classic UCS.

• UCSM can run on classic UCS as well as UCS-Mini.

• UCSM version pre-3.0 cannot be activated on UCS-Mini.

Firmware Management – UCSM Validation

Upgrade Process Demo

UCS Firmware Manual Upgrade Downloading the firmware bundle

Activate UCS Manager

Activate the Sub-ordinate FI

• UCS-A# show cluster state

• UCS-A# connect local-mgmt

• UCS-A (local-mgmt) # cluster {force primary | lead {a | b}}

At this step, do a manual failover to avoid control plane disruption.

Activate the Primary FI

Polling Question 2

Do you use SAN boot for your UCS servers?

1. I use local boot.

2. I use SAN boot for all UCS servers

3. I use local boot but SAN storage for my application Virtual Machines.

4. I use local storage for both boot and application Virtual Machine.

Upgrading Server Firmware

Creating a Host Firmware Package

Simple Configuration

Advanced Configuration

Assign Host Firmware Package to Service Profile

After you save Changes

After the upgrade is successful

Firmware Auto Install

Initiate the Infrastructure Firmware Upgrade

Pending User Ack

Before user ack to reboot the Primary FI

Pending Activities

After the upgrade is complete

Status after the upgrade

Host firmware upgrade using auto install

Select the Firmware Package

Select the Host Firmware Package

Firmware Package Dependencies

Impacted Endpoints

Configuration & Troubleshooting

FC Direct Connect Boot from SAN

Troubleshooting Tools

FC Direct Connect

Enable FC Zoning for each VSANs

Create Storage Connection Policy

Create HBA Initiator Groups

vHBA Initiator Groups

Boot From SAN

Add SAN Boot Target

Associate the boot policy with the Service Profile

Verify that he host can see the target

• UCS-Mini-A# connect adapter 1/3/1

• adapter 1/3/1 # connect

• No entry for terminal type "dumb";

• using dumb terminal settings.

• adapter 1/3/1 (top):1# attach-fls

• No entry for terminal type "dumb";

• using dumb terminal settings.

• adapter 1/3/1 (fls):1# lunlist

• vnic : 14 lifid: 3

• - FLOGI State : flogi est (fc_id 0x470040)

• - PLOGI Sessions

• - WWNN 50:0a:09:83:88:f8:79:f5 WWPN 50:0a:09:83:88:f8:79:f5 fc_id 0x470001

• - LUN's configured (SCSI Type, Version, Vendor, Serial No.)

• LUN ID : 0x0000000000000000 (0x0, 0x4, NETAPP , D1k-Y?E4VkJR) <<<<<<<<<<<<<<

• - REPORT LUNs Query Response

• LUN ID : 0x0000000000000000 <<<<<<<<<<<<<<<<<<

• - Nameserver Query Response

• - WWPN : 50:0a:09:83:88:f8:79:f5

• adapter 1/3/1 (fls):2#

Verify that the LUN is accessible by host

Troubleshooting

• Same tools as classic UCS

• Techsupport

• NXOS (connect nxos)

• Adaptor (connect adaptor)

• Blade (connect cimc)

• All CMC related debug commands will be under NXOS.

• All of the 3.0 UCS-Mini functionality is built upon 2.2.1 UCS Release. For documentation, please refer to:

• http://www.cisco.com/c/en/us/td/docs/unifi

ed_computing/ucs/sw/cli/config/guide/2-2/b_UCSM_CLI_Configuration_Guide_2_2.pdf

• http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-2/b_UCSM_GUI_Configuration_Guide_2_2.pdf

Troubleshooting Tools

Monitoring Session

Span Destination

Select a FI

Create Monitoring Session

Select Scalability Port

Span Source

Select monitoring session

Select Direction Transmit | Recv | Both

Select Scalability Port

Create : eth-traffic-monitoring session

Monitoring Destination

Monitoring Source

eth-traffic-mon session details

Inapplicable Features

Primary Chassis cannot be

decommissioned or removed. (greyed out)

Chassis/FEX Discovery, RACK

server discovery and management connection policy are not applicable. (ignored)

Backup and Restore

• Functionality is identical to classic UCS.

• Import/export files are not UCS-Mini-specific, i.e. file format is platform-independent.

• Files can be imported/exported across UCS-Mini and classic UCS as long as file content (data) is compatible. If incompatible, import operation will fail.

• Full-state backup/restore should not be done across two platforms. But there is no explicit validation for this.

• Recommendation – Do not use import/export across UCS-Mini and classic UCS platforms.

Import / Export

Polling Question 3

Do you use UCS C series server as standalone or integrated to UCSM?

1. I use UCS C series servers in standalone mode.

2. I use UCS C series integrated with UCSM.

3. I don’t use UCS C series server.

Submit Your Questions Now! Use the Q & A panel to submit your questions and our expert will respond

Collaborate within our Social Media

Facebook- http://bit.ly/csc-facebook

Twitter- http://bit.ly/csc-twitter

You Tube http://bit.ly/csc-youtube

Google+ http://bit.ly/csc-googleplus

LinkedIn http://bit.ly/csc-linked-in

Instgram http://bit.ly/csc-instagram

Newsletter Subscription http://bit.ly/csc-newsletter

Learn About Upcoming Events

Cisco has support communities in other languages!

Spanish https://supportforums.cisco.com/community/spanish

Portuguese https://supportforums.cisco.com/community/portuguese

Japanese https://supportforums.cisco.com/community/csc-japan

Russian https://supportforums.cisco.com/community/russian

Chinese

http://www.csc-china.com.cn

If you speak Spanish, Portuguese, Japanese, Russian or Chinese we invite you to participate and collaborate in your language

More IT Training Videos and Technical Seminars on the Cisco Learning Network

View Upcoming Sessions Schedule

https://cisco.com/go/techseminars

Please take a moment to complete the survey

Thank you for Your Time!