cisco support community expert series webcast · tuesday, march 17, 2015 demystifying unified...

Niles Pyelshak

Customer Support Engineer

March 17, 2015

Demystifying Unified Computing System (UCS) Interfaces for troubleshooting

Cisco Support Community

Expert Series Webcast

Upcoming Expert Series Webcast

April 7th, 2015

SourceFire IDS mode integration on ISR G2 and ISR 4K routers will be reviewed in details during the webcast along with a live demo.

SourceFire integration on the ISR G2

and ISR 4K routers with Kureli Sankar

http://bit.ly/webcast-april7




Finesse for UCCX, next generation agent desktop

and transitioning from CAD. Finesse general

capabilities, and how it fits into the UCCX

environment with Abhiram Kramadhati, Cisco

Technical Solutions Manager

Now through March 27th

Ask the Expert Events – Active

Join the discussion for these Ask The Expert Events:

http://bit.ly/knowledgesharing-mar17

Cisco Webex Meeting Server (CWMS):

Install/Configuration and Troubleshooting with Cisco

Experts: Amit Sing, Hari Haran S M, and Jyothi V.




https://supportforums.cisco.com/expert-corner/top-contributors

Participate in Live

Interactive

Technical Events

and much more

http://bit.ly/1jlI93B

Become an Event Top Contributor









Rate Content Now your ratings on documents, videos, and blogs count give points to the authors!!!

So, when you contribute and receive ratings you now get the points in your profile.

Help us to recognize the quality content in the community and make your searches easier. Rate content in the community.

https://supportforums.cisco.com/blog/154746

Encourage and acknowledge people who generously share

their time and expertise



Cisco Support Community Expert Series Webcast

• Today’s featured expert is Cisco Support Engineer, TAC Niles Pyelshak

• Ask your questions now in the Q&A window

Niles Pyelshak Customer Support Engineer

Cisco TAC, Server Virtualization

Tuesday, March 17, 2015

Demystifying Unified Computing

System (UCS) Interfaces for

troubleshooting CCIE#44608

Expert Series Webcast: Demystifying Unified Computing System (UCS) Interfaces for troubleshooting

Technical Expert Question Manager

Robert Burns

CCIE #37856

Now through March 25th

Ask the Expert Event following the Webcast

Join the discussion for these Ask The Expert Events:


Niles will be continuing the discussion in an Ask the Expert

event. So if you have more questions, please visit the

Knowledge Center on the Cisco Support Community

https://supportforums.cisco.com/discussion/12451761/ ask-expert-demystifying-unified-computing-system-ucs- interfaces-troubleshooting




http://bit.ly/webcastate-mar17





















If you would like a copy of the presentation slides, click the PDF file link in the chat box on the right or go to:

bit.ly/docslides-mar17

Thank You For Joining Us Today!




Submit Your Questions Now! Use the Q & A panel to submit your questions

and the panel of experts will respond.

Please take a moment to

complete the survey at

the end of the webcast

Niles Pyelshak

Customer Support Engineer

March 17, 2015

Cisco Support Community Deep Dive Expert Series Webcast

Demystifying Unified Computing System (UCS) Interfaces for troubleshooting

Polling Question 1

Do you have Unified Computing System(UCS) in your Data Center?

a. I have UCS-B series but not C-Series

b. I have UCS-C Series but not B-Series

c. I don’t have UCS and have never used it.

• What’s in a UCS Domain?

• End Host/Switch Mode

• UCS Unified Ports

• Chassis Connectivity

• Server Connectivity

• Traffic Flow

• UCS Interfaces

• HIF to NIF Pinning

• Fabric Failover

• Basic Network Troubleshooting

• Following the Path of a Packet

High Level Overview of UCS Life of a Packet and Troubleshooting

Fabric Interconnect (FI)

• 10 GbE unified fabric switch • Connects up to 160 servers

UCS Fabric Extender (FEX)

• 10 GbE unified remote linecard • Flexible bandwidth connectivity

Compute (Blade and Rack)

• x86 industry standard • Patented extended memory

UCS Virtual Interface Card (VIC)

• CNA adapters • Flexible server I/O

Unified Management

System Components: High-level Overview

UCS Fabric Interconnect

UCS Fabric Interconnect – UCS 6248

• 1RU

• 32 unified base ports and 1 expansion slot • Line rate – 960Gbps

UCS Fabric Interconnect – UCS 6296

• 2RU

• 48 unified base ports and 3 expansion slots • Line rate – 1920 Gbps

UCS Fabric Extender

UCS IO Module (IOM) – 2204 or 2208

• 4 or 8 10GbE fabric links (to Fabric Interconnect)

• 16 or 32 10GbE server links (to servers)

Nexus 2232PP or Nexus 2232TM

• 8 10GbE fabric links (to Fabric Interconnect)

• 32 10GbE server links (to servers)

UCS VIC Adapters

UCS VIC1240 plus Pass-through (PT) Expansion Card - Blades

• VIC1240: Up to 4 x 10 GbE

• PT: Expands VIC1240 up to 8 x 10GbE

• Up to 256 vPCIe

UCS VIC 1280 - Blades

• Up to 8 x 10GE ports • Up to 256 vPCIe

UCS VIC 1225 - Racks

• Up to 2 x 10GE ports • Up to 256 vPCIe

UCS Fabric Portfolio

2 3 4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

CISCO UCS 6296UP

STAT

ID

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

Configuring Unified Ports

• Configured on a per FI basis

• Slider based configuration

• Reboot is required for the new port personality to take into affect

• Recommendation is to configure GEM card, therefore GEM is only needed to be rebooted

Unified Port Screen

Fabric Forwarding Mode of Operations

• End-host mode (EHM): Default mode

• No spanning-tree protocol (STP); no blocked ports

• Admin differentiates between server and network ports

• Using dynamic (or static) server to uplink pinning

• No MAC address learning except on the server ports; no unknown unicast flooding

• Fabric failover (FF) for Ethernet vNICs (not available in switch mode)

• Switch mode: User configurable

• Fabric Interconnects behave like regular ethernet switches

• STP parameters are lock

Fabric Forwarding Mode of Operations Mode of Operations

Putting the Components Together

6200

Fabric A

6200

Fabric B

B200 B250

CNA

I OM A CNA CNA

I OM

B

I OM

A

I OM

B

ETH 1 ETH 2

MGMT MGMT

Chassis 1 Chassis 20

Fabric Switch

Fabric Extenders

Uplink Ports

Compute Blades

Half / Full width

OOB Mgmt

Server Ports

Virtualized Adapters

Cluster

SAN A SAN B

• Configured on a per FI basis

• Slider based configuration

• Reboot is required for the new port personality to take into affect

• Recommendation is to configure GEM card, therefore GEM is only needed to be rebooted

Unified Port Screen

Putting the Components Together

6200

Fabric A

6200

Fabric B

B200 B250

CNA

I OM A CNA CNA

I OM

B

I OM

A

I OM

B

ETH 1 ETH 2

MGMT MGMT

Chassis 1 Chassis 20

Fabric Switch

Fabric Extenders

Uplink Ports

Compute Blades

Half / Full width

OOB Mgmt

Server Ports

Virtualized Adapters

Cluster

SAN A SAN B

UCS Fabric Topologies Chassis Bandwidth Options

24

2x 4 Link 80 Gbps per Chassis

2x 8 Links 160 Gbps per Chassis



2208XP only

• A IOM (sometimes called ‘Fabric Extender’) provides

• A 1GE switch used for internal management (1GE per slot)

• A number of 10G-KR sever facing links (HIF)

• A number of Fabric links (NIF)

• NIC cards on the servers use those HIF ports for external connectivity

• Each IOM provides a separate dedicated IO channel for internal management connectivity

• There is no local switching on IOMs – traffic is always switched by the FIs

What are those IOMs?

Let’s go back in time a bit

26

Chassis Management

Controller

FLASH

EEPROM

DRAM

Control

IO

Chassis Signals

Switch

4 Physical ports to Fabric

Interconnect

8 internal backplane ports to blades

ASIC with some

funny name

Up to 20Gbps

per slot

Up to

80Gbps per

chassis

2104XP

Gen 1 IOM

So this is what we had with the 2104XP

27

2208

2208

220x-XP Architecture

Chassis Management

Controller

FLASH

EEPROM

DRAM

Control

IO

Chassis

Signals

Switch

Woodside ASIC

Internal backplane ports to blades

No Local Switching – ever!

Traffic goes up to FI

Fabric Ports to FI

2204

2204

Feature 2204-XP 2208-XP

ASIC Woodside Woodside

Fabric Ports (NIF)

4 8

Host Ports (HIF)

16 32

CoS 8 8

Latency ~ 500ns ~ 500ns

• These interfaces (show int brief – NXOS shell) are backplane traces

• Eth x/y/z nomenclature where

• x = chassis number

• y = is always 1

• z = host interface port number

Blade Northbound Ports

2

9

Server Connectivity

`

VIC

M81K

R

Palo ASIC Port 0 Port 1

21

04

- B

21

04

- A

B200 M

1/M

2

B 2

30M

1/M

2

x16 Gen1

IOM 2104 with M81KR in M1/M2 Blades

UCS 5100 backplane has 4 passive KR lanes to each half width server slot

BW to half width slot

= Dual 10Gb

IOM 2104 with M81KR in M1/M2 Blades

`

VIC

1280

Sereno ASIC Port 0 Port 1

22

08

- B

22

08

- A

B200 M

1/M

2

B 2

30M

1/M

2

x16 Gen1

IOM 2208 with VIC 1280 in M2 Blades

IOM 2208 has 4 KR lanes to each server slot

BW to half width slot

= Dual 4x10Gb

VIC 1280 has dual 4x10GB ports

IOM 2208 with VIC 1280 in M2 Blades

Traffic Flow

Inter-Fabric Traffic Example (1)

VM1

VM2

VNIC 0

6100 A 6100 B

VNIC 1

L2 Switching

ESX HOST

EHM EHM

Inter-Fabric Traffic Example (2)

VM

1

VM

2

VNIC 0

6100 A 6100 B

VNIC 1

L2 Switching

ESX HOST

EHM EHM

UCS Interfaces

UCS Interfaces

Uplink(border) port

Server(fabric) port

Network(NIF) port

Host(HIF) port

VIF VIF VIF

UCS Fabric

Interconnects

IO Modules

VIC Adapter

Virtual Circuit

Polling Question 1

What are the two types of VIFS?

a) DCE

b) VFC

c) Veth

d) HIF

Virtual Interfaces (VN-TAG)

• Virtual interfaces (VIFs) help distinguish between FC and Eth interfaces

• They also identify the origin server

• VIFs are instantiated on the FI and correspond to frame-level tags assigned to blade mezz cards

• A 6-byte tag (VN-Tag) is preprended by Palo and Menlo as traffic leaves the server to identify the interface

• VN-Tag associates frames to a VIF

• VIFs are ‘spawned off’ the server’s EthX/Y/Z interfaces

VN-Tag: Instantiation of Virtual Interfaces

4

0

Adapter

Switch

10GE

A

Eth 1/1

IOM A

6200-A

Physical Cable

Virtual Cable

(VN-Tag)

Abstracting the Logical Architecture

Blade

10GE

A

vEth 1

IOM A

6200-A

vFC

1

Service Profile

(Server)

Cable

vNIC

1

vEth

1

6200-A

vHBA

1

vFC

1

(Server)

Dynamic, Rapid

Provisioning

State abstraction

Location

Independence

Blade or Rack

Logical Physical

vHBA

1

vNIC

1

42

VN-Tag at the Adapter (Mezz Card) Level

• Ethernet and FC are muxed on the same physical links concept of virtual interfaces (vifs) to split Eth and FC

• Two types of VIFs: veth and vfc

• Veth for Ethernet and FCoE; vfc for FC traffic

• Each EthX/Y/Z or Po interface typically has multiple vifs attached to it to carry traffic to and from a server

• To find all vifs associated with a EthX/Y/Z or Po interface, do this:

VIFs

FCoE: a vfc bound to a veth

Another Way to Find VIFs:

HIF to NIF Pinning

IO Module HIF to NIF Pinning 2208XP – 1 Link

1-4

5-8

9-12

13-16

17-20

21-24

25-28

29-32

FEX Fabric Interconnect

Slot 1

Slot 2

Slot 3

Slot 4

Slot 5

Slot 6

Slot 7

Slot 8

One link


FEX Fabric Interconnect

29-32

1-4

5-8

9-12

13-16

17-20

21-24

25-28

Slot 1

Slot 2

Slot 3

Slot 4

Slot 5

Slot 6

Slot 7

Slot 8

Two links


FEX Fabric

Interconnect

29-32

1-4

5-8

9-12

13-16

17-20

21-24

25-28

Slot 1

Slot 2

Slot 3

Slot 4

Slot 5

Slot 6

Slot 7

Slot 8

Four links


FEX

29-32

1-4

5-8

9-12

13-16

17-20

21-24

25-28

Fabric

Interconnect

Slot 1

Slot 2

Slot 3

Slot 4

Slot 5

Slot 6

Slot 7

Slot 8

Eight links

• What happens in a 4-link topology when you loose 1 link?

• Server interfaces pinned to that link go down*

• The remaining 3 links still pass traffic for the other blade servers

• To recover the failed servers’ vNICs, re-acknowledged of the chassis is required

• After a re-ack UCS falls back to 2 links with regards to blade to fabric port mapping

• That’s because the link count must be a power of 2!

IOM and Failover

* unless you enabled Fabric Failover

HIF to NIF Pinning Port Channel Modes

Port-channel pinning

54

Fabric Port Channeling

Fabric Failover

Polling Question 2

What is the preferred failover?

a) Hardware (Fabric) Failover

b) Software Failover (OS, Hypervisor)

• Fabric Failover is only applicable in EHM.

• NIC teaming software required to provide failover in Switch mode.

Fabric Failover

Recommendation:

End Host Mode

vNIC 1

10GE

10GE

vEth 1

OS / Hypervisor / VM

vEth 1

IOM IOM

PHY Adapter Cisco VIC – M81KR

Menlo – M71KR VIRT Adapter

6100-A 6100-B L1 L2

L1 L2

Physical Cable

Virtual

Cable

Fabric Failover

Following the Path of a Packet

• UCSM top level : show service-profile circuit server <chassis#>/<slot#>

• 2.0(1) release has an improved output


VIF

Virtual Circuit

(VIF 1236)

UCS Fabric

Interconnects

IO Modules

VIC Adapter

Pins out uplink Eth 1/9


VIF

Virtual Circuit

(VIF 1236)

UCS Fabric

Interconnects

IO Modules

VIC Adapter

Interface 1/1/21

VNTAG 2



VIF

Virtual Circuit

(VIF 1236)

UCS Fabric

Interconnects

IO Modules

VIC Adapter

Interface 1/1/21

(Pinned to 1/2)

VNTAG 2


Server Port 1/2



VIF

Virtual Circuit

(VIF 1236)

UCS Fabric

Interconnects

IO Modules

VIC Adapter

VNTAG 2


Designated Receiver Eth 1/9

Interface 1/1/21

(Pinned to 1/2)

Server Port 1/2

C-Series Integration into UCSM

CIMC

GE LOM PCIe Adapter

CPU Mem

OS or Hypervisor

C260M2, C460M2

C220M3, C240M3

C22M3, C24M3 all at CiCM

1.4(6) or higher

C200/C210/C250 NO

Mix of B & C Series is

supported (no B Series

required)

Nexus 2232 Nexus 2232

Mgmt Traffic

Data Traffic

C-Series UCSM Integration Single Wire Management with VIC1225 and UCSM 2.1

UCSM 2.2: Direct Connect with VIC1225 (no FEX)

75

CIMC

GE LOM

CPU Mem

OS or Hypervisor

C260M2, C460M2

C220M3, C240M3

C22M3, C24M3 all at CiCM

1.4(6) or higher

C200/C210/C250 NO

PCIe Adapter

Wrapping up

Polling Question 3

Are B-series servers required to integrate rack mount servers into UCSM?

a) Yes

b) No

• UCS is first and foremost a server, not a switch

• Gen 1 vs Gen 2 components

• 2208 and VIC 1280 allow port-channels

• VIC 1280 with 2208XP for maximum bandwidth

• End-host mode forwarding rules: dynamic pinning

Recap

• Preferred mode of operation should always be end-host mode

• Very much plug and play, scalability, L2 multipathing, fabric failover

• Switch mode: spanning-tree, practically no user configuration possible

• VIF, VN-TAG, HIF, NIFs, VETH, VFC, Border Interfaces

• Operational consistency: C-series integration

Fault Codes: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ts/faults/reference/ErrMess/UCS_SEMs.html

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ts/faults/reference/ErrMess/UCS_SEMs.html



Submit Your Questions Now! Use the Q & A panel to submit your questions and our expert will respond

Collaborate within our Social Media

Facebook- http://bit.ly/csc-facebook

Twitter- http://bit.ly/csc-twitter

You Tube http://bit.ly/csc-youtube

Google+ http://bit.ly/csc-googleplus

LinkedIn http://bit.ly/csc-linked-in

Instgram http://bit.ly/csc-instagram

Newsletter Subscription http://bit.ly/csc-newsletter

Learn About Upcoming Events

http://bit.ly/csc-facebook




http://bit.ly/csc-twitter




http://bit.ly/csc-youtube




http://bit.ly/csc-googleplus




http://bit.ly/csc-linked-in






http://bit.ly/csc-instagram



http://bit.ly/csc-newsletter



Cisco has support communities in other languages!

Spanish https://supportforums.cisco.com/community/spanish

Portuguese https://supportforums.cisco.com/community/portuguese

Japanese https://supportforums.cisco.com/community/csc-japan

Russian https://supportforums.cisco.com/community/russian

Chinese

http://www.csc-china.com.cn

If you speak Spanish, Portuguese, Japanese, Russian or Chinese we invite you to participate and collaborate in your language

https://supportforums.cisco.com/community/spanish



https://supportforums.cisco.com/community/portuguese



https://supportforums.cisco.com/community/csc-japan





https://supportforums.cisco.com/community/russian



http://www.csc-china.com.cn/





More IT Training Videos and Technical Seminars on the Cisco Learning Network

View Upcoming Sessions Schedule

https://cisco.com/go/techseminars

https://cisco.com/go/techseminars

Please take a moment to complete the survey

Thank you for Your Time!

Basic Network Troubleshooting

Check Vmware, Microsoft or UCS

configuration(VLANs, native etc..)

Do you see the VMs

MAC addresses?

Basic Network Troubleshooting: Through Isolation

Isolate two VMs(on

different hosts) or

two bare metal

installs on same the VLAN to the same

FI.

YES

NO

Check local firewall settings on VMs. Arp could be failing. Could

be duplicate IP or MAC issue.

Can you ping

between both VMs?

Isolate VMs to

other fabric and run 3 previous steps

YES

NO

YES

Check

configuration on

switch. Check

for VLANs being trunked over

interfaces, vPC,

STP. UCS disjoint setup.

NO Can you ping the VM

from the upstream L3 device?

Check arp table.

See if you have

duplicate IP or MAC. Check

broadcast

receiver UCS.

NO

YES

Pin VMs out different FI’s. Do you see the

VMs MAC addresses

on upstream switch?

YES

NO Upstream routing

issue.

Can you ping inter-VLAN

between two UCS VMs or

bare metal install?

Basic Network Troubleshooting: Identifying Duplicate IP Addresses

• Disable interface that is

intermittently dropping pings.

• Ping from a VM or L3 device that is in the same VLAN.

• Check to see if MAC address resolved during

ARP. If the MAC did resolve compare/contrast.

• You can disable adapter or shutdown the host. Use MAC address table to see if MAC address moves to other vethernets or interfaces on the upstream switch

• Follow the vethernet or ethernet interface

Basic Network Troubleshooting: Identifying Duplicate MAC Addresses

• Connect nxos : show platform fwm info mac <mac> <vlanID>

• A “Mac history” section was added in 2.0(1) release

• The “old_if_index” is helpful in cases were MAC is learned from different places same time (for example, duplication issue)

Basic Network Troubleshooting: MAC History

VN-Tag at the Adapter (Mezz Card) Level

CIMC

GE LOM PCIe Adapter

CPU Mem

OS or Hypervisor C200M2, C210M2,

C220M3, C240M3,

C250M2, C260M2 or

C460M2

Mix of B & C Series is

supported (no B Series

required)

Nexus 2232 Nexus 2232

2 LOM ports

exclusive CIMC connectivity

Adapter support:

Emulex CNA Qlogic CNA

Intel 10g NIC Broadcom 10g NIC

Cisco VIC

Mgmt Traffic

Data Traffic

C-Series UCSM Integration Before UCSM 2.1 or without VIC1225

96

Let’s go back in time once more

The traces are there; we were just not using them …


• Connect nxos : show platform software enm internal info vlandb id <vlanID>

UCS 6248 Hardware Diagram

98

10 Gig

12 Gig

12 Gig

10 Gig

Carmel 1

Unified Crossbar Fabric Sunnyvale

Carmel 2 Carmel 3

Carmel 4 Carmel 5 Carmel 6

Ca

rme

l

CP

U

Intel

Jasper Forest

PEX 8525

4 port PCIE

Sw itch

South

Bridge

12 Gig

12 Gig

Carmel 1

Carmel cpu

Sunny vale

Carmel 2

Mgmt

Console

Xcon1

Xcon2

DD

R3

x2

PCIE Dual Gig

PCIe x4 PCIe x4 PCIe x4

PCIe x8

PCIE Dual Gig

PCIE Dual Gig

0 1 0 1 N/C

0

1

Carmel 6

Memory

Flash

NVRAM

Serial

• No slot based pinning

• No invalid link count for NIF ports (no “power of 2” rule)

Port-Channel Pinning

9

9

Pinned

to Po

2200-IOM VIC1200

adaptor with

DCE links in

Port-Channel

Gen-1 adaptor

with single 10G

link

8 links, Discrète (default)

slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8

F

E X

8 links, Port-channel

F E

X

Fabric Interconnect

Fabric Interconnect

• Statically pinned to Individual fabric links, no dynamic re-pinning

• Failure of the Fabric port will bring the HIF down triggering a failover

• Statically pinned to logical link (port-channel)

• Failure of the Fabric port will not bring the HIF down no failover take place.

• Traffic re-hashed to available links

HIF to NIF Pinning Link Failure Connectivity from IOM to FI and Failover Behavior

Fabric Failover

Adapter

Switch

10GE

A

Eth 1/1

IOM A

6200-A

Physical Cable

Virtual Cable

(VN-Tag)

Abstracting the Logical Architecture

102

Blade

10GE

A

vEth 1

IOM A

6200-A

vFC

1

Service Profile

(Server)

Cable

vNIC

1

vEth

1

6200-A

vHBA

1

vFC

1

(Server)

Dynamic, Rapid

Provisioning

State abstraction

Location

Independence

Blade or Rack

Logical Physical

vHBA

1

vNIC

1

cisco support community expert series webcast · tuesday, march 17, 2015 demystifying unified...

Documents