hpe mellanox switch training - home | iotlabs...pptv network diagram 3 link aggregation data vlan...

HPE Mellanox Switch TrainingSuchart BoonpanMASE, CCNP, ACE-A, ACMP

May 2020

Network Diagram

Confidential – For Training Purposes Only 2

PPTV NETWORK DIAGRAM

3

Link Aggregation

Data VLAN 300

ILO

Cisco core L3 #1, #2

SN2410M

Cisco Mgmt/iLO switch

VLAN 400

Mgmt0, 1G

iLO

Qumulo

Servers 1-6

MLAG Mgmt IPMgmt0 SW1: 10.0.99.61Mgmt0 SW1: 10.0.99.62MLAG VIP: 10.0.99.63

MAGP 2:Interface vlan: 400State : MasterVirtual IP : 10.0.102.1SW1 IP : 10.0.102.10/24Sw2 IP : 10.0.102.10/24Virtual MAC : 00:00:5E:00:01:02

inter-peer link (IPL) Management Plane (Active/Standby)

SN2410MMgmt0, 1G

P49-50100GMpo1-6

P1-6 25G

Mpo30P24, 10G

MAGP 1:Interface vlan: 300State : MasterVirtual IP : 10.0.1.254SW1:IP: 10.0.1.252/23SW2:IP: 10.0.1.253/23Virtual MAC : 00:00:5E:00:01:01

4

Cost-optimized unique form factors Optimized for storage performanceFlexible pricing & investment protection

SN2010M SN2100M SN2410bM SN2410M SN2700M/SN2745M

18 Port 8 / 16 Port 24 / 48 Port 24 / 48 Port 16 / 32 Port

Half Width Half Width Full Width Full Width Full Width / Full & Short Depth

18 x 1/10/25G SFP28 + 4 x 40/100G QSFP28

16 x 40/100 QSFP28 48 x 10G SFP + 8 x 40 QSFP

48 x 10/25G SFP28 + 8 x 40/100G QSFP

32 x 40/100G QSFP28

34 x 10/25G SFP28 64 x 10/25 SFP28 64 x 10GbE SFP 64 x 10/25G SFP28 64 x 10/25G SFP28

HPE M-SERIES SWITCHES SPECIFICATION

5

DOWNLINKS (COMPUTE AND STORAGE CONNECT)HPE M‐Series SN2010M

(18) 10/25Gb SFP28 Ports(4) 10/25/40/100Gb QSFP28 Ports

25G

25G

25G SFP+ DAC

25Gb SFP+ Direct Attached Copper (DAC .5m to 3m)HPE 25Gb SFP28 to SFP28 0.5m Direct Attach Copper Cable (844471-B21)

HPE 25Gb SFP28 to SFP28 1m Direct Attach Copper Cable (844474-B21)

HPE 25Gb SFP28 to SFP28 3m Direct Attach Copper Cable (844477-B21)

6

CROSSLINKS (MLAG 100G PREFERRED BEST PRACTICE)

HPE M‐Series SN2100M(16) 10/25/40/100Gb QSFP28 Ports

100GHPE M‐Series SN2010M(18) 10/25Gb SFP28 Ports

(4) 10/25/40/100Gb QSFP28 Ports

100G QSFP28 DAC

100G

100G

100G

100Gb QSFP28 Direct Attached Copper (DAC 1m)HPE X240 100G QSFP28 to QSFP28 1m Direct Attach Copper Cable (JL271A)

7

M-SERIES SN2410M ETHERNET SWITCH

8

SN2410M M-SERIES NETWORK CONNECTIONS

25 Gb DAC cables

1 Gb UTP

1Gb, 10Gb, 25 Gb uplink (with

matching transceiver or adapter) 100 Gb DAC cables

10 Gb DAC cables

Q3

Q1

Q2

Q4

MLAG 1

MLAG 2Edge 2

Edge 1

MLAG 1

MLAG 2

MGMT

MGMT

Q1 iLO

Q2 iLO

Q3 iLO

Q4 iLO

Tor1 Mgmt

Tor2 Mgmt

Q ports go to the 640SFP28 NIC in the Qumulo Nodes.

Comp1

Comp2

Comp ports go to the 640SFP28 NIC in the Compute Nodes.

Comp3 iLO

Comp4 iLO

Comp1 iLO

Comp2 iLO

Comp3

Comp4

Q3

Q1

Q2

Q4

Edge 2

Edge 1

Comp1

Comp2

Comp3

Comp4

Q ports go to the 640SFP28 NIC in the Qumulo Nodes.

Comp ports go to the 640SFP28 NIC in the Compute Nodes.

Q&A

– Question ?

– Break 5 m

9

Mellanox Switch User Interfaces


User Interfaces of the MLNX-OS®

1. Command Line Interface (CLI)

The CLI is accessed through: SSH, Telnet sessions, or directly via the console port on the front pane

2. Web Interface (web GUI)

The Web GUI is accessed through: HTTP or HTTPS

11

Connection with MLNX-OS

1. Access the MLNX-OS CLI via Serial Console.

12

User name and password

3. Log in with default credentials.– User name: admin– Password: admin


login as: admin

Mellanox MLNX-OS Switch Management

Using keyboard-interactive authentication.Password:admin


2. Access the MLNX-OS WebUI via HTTP/HTTPS

14

Network

Mellanox Switch

PC


3. Access the MLNX-OS CLI via SSH

15

Network

Mellanox Switch

PC

CLI configuration modes—Standard and enable

Standard– Identified by the CLI prompt >.– Most restrictive mode.– Includes commands that query only a restricted set of state information.– Users cannot take any actions that directly affect the system—like rebooting the switch, or changing the

configuration.– Use the enable command to move from standard mode to enable mode.

Enable– Identified by the CLI prompt #.– Offers commands to view all state information, and take actions like rebooting the system.– Does not allow you to change any configurations.

– Use the disable command to move from enable mode to standard mode.


switchA [standalone: master] > enable

switchA [standalone: master] # disable

Global configuration mode—Config

Config

– Identified by CLI prompt (config)#.

– Allowed only for user accounts in the “admin” role.

– Has a full, unrestricted set of commands to view anything, take any action, and change any configuration.– Use the configure terminal command to move from enable mode to config mode.

– Use ‘exit’ command to move from config mode to enable mode.


g1switchA [standalone: master] # configure terminalg1switchA [standalone: master] (config) #

g1switchA [standalone: master] (config) # exitg1switchA [standalone: master] #

Getting help

– Use ‘?’ from any mode to view available commands.Use the space bar to see more commands, or ‘q’ to quit the display.

– Use the command followed by ‘?’ to view available command parameters.


switchA [standalone: master] (config) # ?aaa Configure Authentication, Authorization, and Accountingaccess-list Configure access-list actionbanner Set system bannersboot Configure system boot parametersclear Reset certain statistics or clear cachescli Configure CLI shell optionsclock Set the system clock or timezoneconfiguration Manipulate configuration files

switchA [standalone: master] (config) # show ?aaa Display Authentication, Authorization, and Accounting settingsaccess-list Display IPV4 informationaccess-lists List access listsasic-version Display asic versionbanner Display banner settingsbootvar Display installed system images and boot parameterscli Display CLI optionsclock Display system time and date

CLI commands autocomplete

– Use [Tab] to auto-complete commands.c [Tab] - displays all commands that start with ‘c.’

– co [Tab] - autocompletes to ‘configure.’

– Unique prefix of a command can be used, instead of the full command.Example: ‘co t’ can be used instead of ‘configure terminal.’


switchA [standalone: master] # cclear cli configure crypto

switchA [standalone: master] # co tswitchA [standalone: master] (config) #

Saving the configuration

– Save running-config into active-config.

Or


switchA [standalone: master] (config) # configuration writeswitchA [standalone: master] (config) # show configuration filesinitial (active)initial.bak

Active configuration: initialUnsaved changes: no

switchA [standalone: master] (config) # write memory

Mellanox Switch Image (Operating System)


MLNX-OS images

– Two images are stored in the flash memory: Partition 1 and Partition 2.

– By default, the image from Partition 1 is loaded at reboot.

– Mellanox Operating system = ONYX = MLNX-OS (same thing)


switchA [standalone: master] (config interface mgmt1) # show images

Installed images:Partition 1:

version: X86_64 3.8.2204 2019-12-29 16:11:11 x86_64

Partition 2:version: X86_64 3.7.1134 2019-01-24 13:38:57 x86_64

Last boot partition: 1Next boot partition: 1

WebUI Onyx Image Upgrade


1. Use the IP address of each Switch of the management interface on the address bar of your browser, example;

(switchA 10.25.19.11)(switchB 10.25.19.12)

2. Type in user name and password default: admin, admin

3. Press Login



1. Choose System

2. Choose Onyx Upgrade

3. Select Install from local file:

4. Select Choose file:

5. Select Install Image



1. Choose System



4. Select Choose file: onyx-X86_64-3.8.2204




1. Choose System





WebUI Update Status


1. Please note file copy and then Install

2. Please note Image Update Status

3. Please select Reboot

WebUI Update Status





Q&A

– Question ?

– Break 10 m or Lunch

32

MLAG – Multi Chassis LAG


List of Network protocols used in this project.

34

– MLAG L2

– MLAG L3 (MAGP)

– Interface Port-Channel (Link Aggregation)

– Interface MLAG Port-Channel (Multi Chassis Link Aggregation)

– Spanning tree mode RPVST

– Switch port mode Hybridge– Static Route

MLAG – Multi Chassis LAG:

– Physical ports of two separate switches are aggregated in one logical port.

– MLAG switches appear as a single Layer 2 switch.

– A peering device (host or switch) runs a standard LAG, and is not aware of the fact that its LAG is connected to two separate switches.

– MLAG provides:

– High bandwidth and load-balancing

– High availability in case of a link failure

– High availability in case of a switch failure or a switch software upgrade

Layer 3 Network

LAG

MLAG

IPL

Qumulo 1

switchA switchB

Layer 3 Network

The MLAG protocol components.

36

• Keepalive

• Unicast and multicast sync• MLAG port sync

MLAG Keepalive and Failover

37

– Master election in MLAG is based on the highest IPL VLAN interface IPs of the nodes.

– The MLAG pair of switches periodically exchanges a keepalive message (via IPL)

– If the keepalive message fails to arrive for three consecutive intervals the switches break into two standalone switches.

– If IPL fail, the slave shuts down its interfaces to avoid a split brain scenario and the master becomes a standalone switch.

Unicast and Multicast Sync

38

– It prevents unicast asymmetric traffic from loading the network with flood traffic

MLAG Port Sync

39

– Under normal circumstances, traffic from the IPL cannot pass through the MLAG ports (the IPL is isolated from the MLAG ports).

– If one of the MLAG links break, the other MLAG switch opens that isolation and allows traffic from its peer through the IPL to flow via the MLAG port which accesses the destination of the fallen link.

1. Enable IP routing – MLAG may be enabled without IP routing, but without IP routing an IPL VLAN interface cannot be

configured and thus MLAG does not function.2. Enable IGMP snooping

– MLAG may be enabled without IGMP snooping, but if IGMP snooping is disabled, multicast FDBs do not synchronize.

3. Enable LACP – if dynamic LAG is used.4. Enable MLAG protocol commands.

– g1switchA configuration:

– g1switchB configuration:

MLAG Global Configurations

g1switchA [standalone: master] (config) # ip routingg1switchA [standalone: master] (config) # ip igmp snoopingg1switchA [standalone: master] (config) # lacpg1switchA [standalone: master] (config) # protocol mlag

g1switchB [standalone: master] (config) # ip routingg1switchB [standalone: master] (config) # ip igmp snoopingg1switchB [standalone: master] (config) # lacpg1switchB [standalone: master] (config) # protocol mlag

1. Create a port-channel:

– Port-channel indexes on two switches may differ.

2. Set the port-channel as an IPL.

3. Group physical ports to the port-channel.

– LACP or static LAG can be used.

– switchA configuration:

– switchB configuration:g1switchB [standalone: master] (config) # interface port‐channel 1g1switchB [standalone: master] (config interface port‐channel 34) # ipl 1g1switchB [standalone: master] (config interface port‐channel 34) # exitg1switchB [standalone: master] (config) # interface ethernet 1/19‐1/20g1switchB [standalone: master] (config interface ethernet 1/19‐1/20) # channel‐group 1 mode activeg1switchB [standalone: master] (config interface ethernet 1/19‐1/20) # exit

g1switchA [standalone: master] (config) # interface port‐channel 1g1switchA [standalone: master] (config interface port‐channel 1) # ipl 1g1switchA [standalone: master] (config interface port‐channel 1) # exitg1switchA [standalone: master] (config) # interface ethernet 1/19‐1/20g1switchA [standalone: master] (config interface ethernet 1/19‐1/20) # channel‐group 1 mode activeg1switchA [standalone: master] (config interface ethernet 1/19‐1/20) # exit

IPL Port-Channel

IPL VLAN Configuration

1. Create a VLAN and a VLAN interface for the IPL.

2. Set an IP address and a netmask for the VLAN interface.– The switch with highest IP address is elected as the MLAG master.

3. Map the VLAN interface to be used on the IPL and set the peer’s IP address.

–switchA configuration:

– switchB configuration:

g1switchA [standalone: master] (config) # vlan 4094g1switchA [standalone: master] (config vlan 4094) # exitg1switchA [standalone: master] (config) # interface vlan 4094g1switchA [standalone: master] (config interface vlan 4094) # ip address 172.16.34.253 /30g1switchA [standalone: master] (config interface vlan 4094) # ipl 1 peer‐address 172.16.34.254g1switchA [standalone: master] (config interface vlan 4094) # exit

g1switchB [standalone: master] (config) # vlan 4094g1switchB [standalone: master] (config vlan 4094) # exitg1switchB [standalone: master] (config) # interface vlan 4094g1switchB [standalone: master] (config interface vlan 4094) # ip address 172.16.34.254 /30g1switchB [standalone: master] (config interface vlan 4094) # ipl 1 peer‐address 172.16.34.253g1switchB [standalone: master] (config interface vlan 4094) # exit

1. Configure the MLAG cluster:

– Both switches must be configured with an identical unique group name.

– One of the switches is also configured with the VIP –it is the cluster master.

– VIP address must be of the management subnet.


– switchB configuration:

– Wait for a few seconds until prompt is changed to group name and cluster master/standby.

MLAG Cluster Configuration

switchA [standalone: master] (config) # mlag‐vip MLAG‐G1 ip 10.25.19.13 /16switchA [MLAG‐G1: master] (config) #

switchB [standalone: master] (config) # mlag‐vip MLAG‐G1 switchB [MLAG‐G1: standby] (config) #

– Enable MLAG protocol:

– MLAG protocol is disabled by default.


– switchB configuration

switchA [MLAG‐G1: master] (config) # mlagswitchA [MLAG‐G1: master] (config mlag) # no shutdown

Enable MLAG Protocol

switchB [MLAG‐G1: standby] (config) # mlagswitchB [MLAG‐G1: standby] (config mlag) # no shutdown

Verify MLAG VIP Configuration

switchA [MLAG‐ACAD: master] (config) # show mlag‐vipMLAG VIP========MLAG group name: MLAG‐G1MLAG VIP address: 10.25.19.13/16Active nodes: 2

Hostname VIP‐State IP Address‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐switchA master 10.25.19.11switchB standby 10.25.19.12

Cluster master/ standby

switchA [MLAG‐G1: master] (config) # show mlagAdmin status: EnabledOperational status: UpReload‐delay: 30 secKeepalive‐interval: 1 secUpgrade‐timeout: 60 minSystem‐mac: 00:00:5E:00:01:57

MLAG Ports Configuration Summary:Configured: 1Disabled: 0Enabled: 1

MLAG Ports Status Summary:Inactive: 0Active‐partial: 0Active‐full: 1

MLAG IPLs Summary:ID Group Vlan Operational Local Peer

Port‐Channel Interface State IP address IP address‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐1 Po1 34 Up 172.16.34.253 172.16.34.254

MLAG Members Summary:System‐id State Hostname‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐00:02:C9:A8:E2:D8 Up <g1switchA>00:02:C9:83:84:48 Up g1switchB

Verify MLAG Configuration

Switch g1switchA ‐ MLAG master

MLAG virtual MAC

Switch g1switchB ‐ MLAG Standby

1. Create a mlag-port-channel: Host Configuration– ‘mlag-port-channel’ indexes must be identical on both

MLAG switches.

– LACP or static LAG can be used.

2. Qumulo Node physical ports to the mlag-port-channel.


– switchB configurationswitchB [MLAG‐G1: standby] (config) # interface mlag‐port‐channel 16switchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # exitswitchB [MLAG‐G1: standby] (config) # interface ethernet 1/16switchB [MLAG‐G1: standby] (config interface ethernet 1/16) # mlag‐channel‐group 16 mode activeswitchB [MLAG‐G1: master] (config interface ethernet 1/16) # exit

switchA [MLAG‐G1: master] (config) # interface mlag‐port‐channel 16 switchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # exitswitchA [MLAG‐G1: master] (config) # interface ethernet 1/16switchA [MLAG‐G1: master] (config interface ethernet 1/16) # mlag‐channel‐group 16 mode activeswitchA [MLAG‐G1: master] (config interface ethernet 1/16) # exit

MLAG Port-Channel Creation

1. Disable STP for the mlag-port-channel:

2. Enable mlag-port-channel:

– Default admin state of mlag-port-channel is disabled.In order to allow administrator to configure bothswitches first, and then enable MLAG.


– switchB configuration

switchA [MLAG‐G1: master] (config) # interface mlag‐port‐channel 16switchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # spanning‐tree port type edgeswitchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # spanning‐tree bpdufilter enableswitchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # no shutdown

MLAG Port-Channel Configuration

switchB [MLAG‐G1: standby] (config) # interface mlag‐port‐channel 16switchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # spanning‐tree port type edgeswitchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # spanning‐tree bpdufilter enableswitchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # no shutdown

– Physical ports flags:– Down - port is down– Up – ports is up

– ‘mlag-port-channel’ flags:– Partial Up – local or remote are down – Up – both local and remote are up– Down – ‘admin’ state is disabled

Verify mlag-port-channel Configuration

switchA [MLAG‐G1: master] (config) # show interfaces mlag‐port‐channel summaryMLAG Port‐Channel Flags: D‐Down, U‐Up

P‐Partial UP, S ‐ suspended by MLAGPort Flags: D ‐ Down, P ‐ Up in port‐channel (members)

S ‐ Suspend in port‐channel (members), I ‐ Individual

GroupPort‐Channel Type Local Ports Peer Ports(D/U/P/S) (D/P/S/I) (D/P/S/I)‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐1 Mpo16(U) LACP Eth1/16(P) Eth1/16(P)

IMPORTANT: Please take note, links will be Down until the Qumulo node adapter ports are set to support LACP

Q&A

– Question ?

– Break 10 m

50

Virtual local area network (VLAN)


Virtual local area network (VLAN)

– A virtual local area network (VLAN) is a logical segment of the Ethernet network that defines a broadcast domain.

– A VLAN is identified by a VLAN ID.

– Each VLAN should be assigned with a unique IP subnet.

– Hosts within the same VLAN can communicate with each other in layer 2.

– Hosts in different VLANs can communicate with each other in layer 3.


VLAN1

VLAN2 VLAN3

VLAN1

VLAN2VLAN3

VLAN1VLAN2VLAN3

Trunk ports

– Trunk ports carry traffic for multiple VLANs across a single link.

– Hosts in the same VLAN, that are connected to different switches, can communicate with each other over the trunk link.

– When a frame is sent on the trunk port, the sending switch adds a tag that contains the VLAN ID.The receiving switch reads the VLAN ID and removes the tag.


VLAN1

VLAN2 VLAN3

VLAN1

VLAN2VLAN3

VLAN1VLAN2VLAN3

trunk port

IEEE 802.1Q trunking protocol

– The IEEE 802.1Q trunking protocol defines the tag added to Ethernet frames carried over a trunk port.

– The 802.1Q tag is 4 bytes in size, including the 12-bit VLAN ID.


Original Ethernet frame6 bytes 6 bytes 2 bytes 46-1500 bytes 4 bytes

DestinationMAC

Source MAC

Type/Length

Data FCS

6 bytes 6 bytes 4 bytes 2 bytes 46-1500 bytes 4 bytes

DestinationMAC

Source MAC

802.1Q Tag Type/ Length Data New FCS

802.1Q tagged Ethernet frame

2 bytes 3 bits 1 bit 12 bits

Type Priority CFI VLAN ID

Switch port types

A switch port can be configured in access, trunk, or hybrid mode.

– Access– The port accepts and sends only untagged frames– Frames are assigned to the configured port VLAN ID (PVID)– Usually, an access port is connected to a host

– Trunk– The port accepts and sends only tagged frames– Untagged frames are dropped– Usually, a trunk port is connected to another switch

– Hybrid– The port accepts and sends both tagged and untagged frames– Untagged frames are assigned to the configured port VLAN ID (PVID)– A hybrid port is connected to either a switch or a host


Configure new VLAN’s

– switchA Configuration.

* VLANs 1-4094 are supported.

– switchB Configuration


switchA [MLAG-G1: master] (config) # vlan 507switchA [MLAG-G1: master] (config vlan 507) # exitswitchA [MLAG-G1: master] (config) # vlan 400switchA [MLAG-G1: master] (config vlan 400) # exitswitchA [MLAG-G1: master] (config) #

switchB [MLAG-G1: standby] (config) # vlan 507switchB [MLAG-G1: standby] (config vlan 507) # exitswitchB [MLAG-G1: standby] (config) # vlan 400switchB [MLAG-G1: standby] (config vlan 400) # exitswitchB [MLAG-G1: standby] (config) #

Configure VLAN Interface and IPV6




switchA [MLAG-G1: master] (config) # interface vlan 507 ip address 192.168.17.3 /24 switchA [MLAG-G1: master] (config) # ipv6 routingswitchA [MLAG-G1: master] (config) # interface vlan 1 ipv6 enableswitchA [MLAG-G1: master] (config) # interface vlan 507 ipv6 enable

switchB [MLAG-G1: standby] (config) # interface vlan 507 ip address 192.168.18.3 /24switchB [MLAG-G1: standby] (config) # ipv6 routingswitchB [MLAG-G1: standby] (config) # interface vlan 1 ipv6 enableswitchB [MLAG-G1: standby] (config) # interface vlan 507 ipv6 enable

Verify new VLANs

– Verify new VLANs configuration.

– VLAN 1 is the default VLAN, and all ports are assigned to it.


switchA [MLAG-G1: master] (config) # show vlan----------------------------------------------------------------------VLAN Name Ports----------------------------------------------------------------------1 default Eth1/1, Eth1/2, Eth1/3, Eth1/4, Eth1/5,

Eth1/6, Eth1/7, Eth1/8, Eth1/9, Eth1/10,Eth1/11, Eth1/12, Eth1/13, Eth1/14, Eth1/15,Eth1/17, Eth1/18, Eth1/21, Eth1/22, Mpo16

4005074094

Assign VLAN’s to host Interfaces




switchA [MLAG-G1: master] (config) # interface ethernet 1/16 description Qumulo-Node1 switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport mode hybridswitchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport access vlan 1switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport hybrid allowed-vlan 507

switchB [MLAG-G1: standby] (config) # interface ethernet 1/16 description Qumulo-Node1switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport mode hybridswitchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport access vlan 1switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport hybrid allowed-vlan 507

Verify VLANs Assignment

– Verify new VLANs configuration.

– VLAN 1 is the default VLAN, and all ports are assigned to it.


switchA [MLAG-G1: master] (config) # show vlan----------------------------------------------------------------------VLAN Name Ports----------------------------------------------------------------------1 default Eth1/1, Eth1/2, Eth1/3, Eth1/4, Eth1/5,

Eth1/6, Eth1/7, Eth1/8, Eth1/9, Eth1/10,Eth1/11, Eth1/12, Eth1/13, Eth1/14, Eth1/15,Eth1/17, Eth1/18, Eth1/21, Eth1/22, Mpo16

400507 Mpo164094

Configuring Access Mode and Assigning Port VLAN ID (PVID)

61

switch > enableswitch # configure terminalswitch (config) # vlan 6switch (config vlan 6) #switch (config vlan 6) # exitswitch (config) #switch (config) # interface ethernet 1/22switch (config interface ethernet 1/22) #switch (config interface ethernet 1/22) # switchport mode accessswitch (config interface ethernet 1/22) # switchport access vlan 6switch (config 1/22) # exitswitch (config) #

Configuring Hybrid Mode and Assigning Port VLAN ID (PVID)

62

switch > enableswitch# configure terminalswitch (config) # vlan 6switch (config vlan 6) #switch (config vlan 6) # exitswitch (config) #switch (config) # interface ethernet 1/22switch (config interface ethernet 1/22) #switch (config interface ethernet 1/22) # switchport mode hybridswitch (config interface ethernet 1/22) #switch (config interface ethernet 1/22) # switchport access vlan 6switch (config interface ethernet 1/22) #switch (config interface ethernet 1/22) # exitswitch (config) #

Configuring Trunk Mode VLAN Membership

63

switch > enable

switch # configure terminal

switch (config) # vlan 10

switch (config vlan 10) #

switch (config vlan 10) # exit

switch (config) #

switch [standalone: master] (config) # interface ethernet 1/35

switch [standalone: master] (config interface ethernet 1/35) #

switch [standalone: master] (config interface ethernet 1/35) # switchport mode trunkswitch [standalone: master] (config interface ethernet 1/35) #

Q&A

– Question ?

– Break 10 m

64

Spanning Tree Protocol (STP)


Multiple spanning tree (MST)

66

- MST maps multiple VLANs to an instance, reducing the number of spanning-tree instances.

- MST and PVST+ are compatible

- Backwards compatible with RSTP and STP- It is the IEEE standard protocol (802.1s)

Rapid spanning tree (RSTP)

67

- The Rapid Spanning Tree Protocol recovers (converges to a new spanning tree) more quickly than STP

- It is backwards-compatible with MST and STP.- It is the IEEE standard protocol (802.1w)

Rapid per-VLAN spanning tree (RPVST)

68

- Cisco proprietary version of Rapid Spanning Tree Protocol (802.1w)

- It creates a spanning tree for each VLAN, just like PVST.- Rapid-PVST is backward compatible with standard Per-VLAN Spanning Tree (PVST/802.1d)

Ethernet layer 2 loops

– Layer 2 redundant links are required to provide a backup path in case of link or switch failure.

– Redundant links result in layer 2 loops—There are multiple paths between a pair of nodes.

– Layer 2 loops cause “broadcast storms.”– When an Ethernet broadcast frame is sent in the network, it

endlessly circulates in a loop consuming all available bandwidth.

– Broadcast storms deny bandwidth for normal network traffic.


Spanning Tree Protocol (STP)

– Spanning Tree Protocol (STP) is an IEEE 802.1D standard.

– STP ensures a loop-free topology for Ethernet networks.

– STP allows a network design to include redundant links and to provide automatic backup paths, if an active link fails.

– STP identifies redundant links and puts redundant ports in blocking state.

– When a topology change occurs, STP reacts and moves blocked ports to the forwarding state.

– Convergence time is 30 to 50 seconds.


X X

Rapid Per-VLAN Spanning Tree (RPVST) Configuration


Configure Spanning Tree RPVST




switchA [MLAG-G1: master] (config) # spanning-tree mode rpvst

switchB [MLAG-G1: standby] (config) # spanning-tree mode rpvst

Configure Flowcontrol and Jumbo Frames




switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 flowcontrol receive on forceswitchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 flowcontrol send on forceswitchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 mtu 9216 force

switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 flowcontrol receive on forceswitchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 flowcontrol send on forceswitchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 mtu 9216 force

Maximum Transmission Unit (MTU) Size

74

– The largest possible frame size of a communications Protocol Data Unit (PDU) on an OSI Model Layer 2 data network.– Default frame size is 1518 bytes– Example of commands to check MTU size;

C:\Users\ScottHogg> ping 192.168.10.1 -l 1500 –f

RedHat# ping -s 1500 -M do 192.168.10.1

Router1# ping 192.168.10.1 size 1500 df-bit

Switch7K# ping 192.168.10.1 packet-size 9216 c 10

RP/0/RP0/CPU0:Router1#ping 192.168.10.1 size 1500 donnotfrag

Junos-root@J4350-1# run ping 192.168.10.1 size 1500 do-not-fragment rapid

Q&A

– Question ?

– Break 10 m

75

Link Layer Discovery Protocol (LLDP)


Link Layer Discovery Protocol (LLDP)

– Link Layer Discovery Protocol (LLDP) is a vendor-neutral protocol defined in IEEE 802.1AB.

– LLDP is used by network devices for advertising their identity, capabilities, and neighbors on an IEEE 802 LAN.

– LLDP can be used to discover and verify Ethernet network topology.

– LLDP is by default globally disabled.

– LLDP frames are sent every 30 seconds by all LLDP enabled interfaces.


Eth1/1

switchA switchB

Configure LLDP




switchA [MLAG-G1: master] (config) # lldp

switchB [MLAG-G1: standby] (config) # lldp

Show local LLDP information – Show local LLDP information.

– Show interface LLDP information.


switchA [MLAG-G1: master] (config) # show lldp localLLDP: enabled

Local global configurationChassis sub type: Mac AddressChassis id: b8:59:9f:70:d6:00System Name: g1switchASystem Description: SN2010M,Onyx,SWv3.8.2204Supported capabilities: B,RSupported capabilities enabled: B

switchA [MLAG-G1: master] (config) # show lldp interfaces ethernet 1/19TLV flags

PD: port-descriptionSN: sys-nameSD: sys-descriptionSC: sys-capabilitiesMA: management-addressETS-C: ETS-Configuration

Interface Receive Transmit TLVs-----------------------------------------------------------------------------------Eth1/19 Enabled Enabled PD, SN, SD, SC, MA, PFC, AP, ETS-C, ETS-R

Eth1/19

switchA switchB

View Cable/Transceiver – Show local Transceivers information.


switchA [MLAG-G1: master] (config) # show interfaces ethernet 1/19 transceiverPort 1/19 state

identifier : QSFP28cable/module type : Passive copper cableethernet speed and type: 100GBASE-CR4vendor : Mellanoxcable length : 1mpart number : 845404-B21revision : A1serial number : 6C2749003C

Eth1/19

switchA switchB

Onyx system features


Feature Description

Software management – Dual software image– Software and firmware updates

File management – FTP, TFTP, SCP

Logging – Event history log– Syslog support

Chassis management – Monitoring environmental controls– Power management– Auto-temperature control– High availability

Network management interfaces – SNMP v1,v2c,v3– Puppet agent

Security – SSH, Telnet– RADIUS, TACACS+

Date and time – NTP

Cables and transceivers – Transceiver info

Configure Clock and NTP




switchA [MLAG-G1: master] (config) # clock timezone America North United_States CentralswitchA [MLAG-G1: master] (config) # clock set 09:00:00 2020/05/12switchA [MLAG-G1: master] (config) # ntp server 10.187.2.2switchA [MLAG-G1: master] (config) # ntp enable

switchB [MLAG-G1: master] (config) # clock timezone America North United_States CentralswitchB [MLAG-G1: master] (config) # clock set 09:00:00 2020/05/12switchB [MLAG-G1: master] (config) # ntp server 10.187.2.2switchB [MLAG-G1: master] (config) # ntp enable

Show NTP and Clock – Display commands NTP Status.

– Display Clock settings.


NTP is administratively : enabledNTP Authentication administratively: disabledNTP server role : enabledClock is synchronized:Reference: 10.187.2.2Offset : -0.620 ms

Active servers and peers:10.187.2.2:Conf Type : servStatus : sys.peer(*)Stratum : 1Offset(msec) : -0.620Ref clock : .GPS.Poll Interval (sec): 64Last Response (sec): 53Auth state : none

switchA [MLAG-G1: master] (config) # show clockTime: 14:54:50Date: 2020/05/12Time zone: America North United_States Central (US/Central)UTC offset: -0500 (UTC minus 5 hours)

Reset factory defaults– Reset the switch to factory defaults.


g1switchA [MLAG-G1: master] (config) # reset factory ?<cr>halt Halt system after reset, instead of rebootingkeep-all-config Preserve all configuration files (supercedes keep-basic)keep-basic Preserve licenses in the active configurationkeep-virt-vols Preserve all virtual disk volumesonly-config Reset only configurationmseries3 [standalone: master] (config) # reset factory keep-all-configWarning - confirming will cause system reboot.Type 'YES' to confirm reset:

Configuration Licenses System profile Management interfaces

keep-all-config Unchanged Not deleted Unchanged Unchanged

keep-basic Reset Not deleted Reset Reset

only-config Reset Deleted Reset Unchanged

Reset Factory




switchA [MLAG-G1: master] (config) # reset factory

switchB [MLAG-G1: standby] (config) # reset factory

Q&A

– Question ?

– Break 10 m

86

Multi-active gateway protocol (MAGP)


Enable the switch as host’s gateway (Layer 3 MLAG)

88

There are two protocols of the Layer 3 MLAG;- VRRP: Virtual Router Redundancy Protocol.

It’s working as Active/Standby.

- MAGP: Multi-active gateway protocol.

It’s working as Active/Active.Note: MAGP is the recommend protocol for implementing Mellanox’s L3 MLAG.

Multi-active gateway protocol (MAGP)

89

- To solve the default gateway problem when a host is

connected to a set of switch routers via MLAG.

- Each switch routers is an active default gateway router

to the host.

- Directly forwarding IP traffic to the L3 cloud regardless

which SR traffic comes through.

Configuring MAGP Example

90

- switch (config)# ip routing

- switch (config)# vlan 20

- switch (config)# interface vlan 20

- switch (config interface vlan 20)# ip address 11.11.11.11 /8

- switch (config interface vlan 20)# no shutdown

- switch (config)# protocol magp

- switch (config interface vlan 20)# magp 100

- switch (config interface vlan 20 magp 100)# ip virtual-router address 11.11.11.254- switch (config interface vlan 20 magp 100)# ip virtual-router mac-address AA:BB:CC:DD:EE:FF

Verify the MAGP configuration

91

IP Routing

92

– IP Interfaces (L3)

– MLNX-OS supports the following 3 types of IP interfaces:

•VLAN interface

•Loopback interface

•Router port interface

Note: Router port interfaces are not supported on SX10xx-xxxR and SX60xx-xxxR systemsNote: Routing for this project is using VLAN interface with ip route 0.0.0.0 0.0.0.0 10.0.102.2

VLAN interface Attributes

– VLAN interface is a logical IPv4 interface created per subnet over a specific 802.1Q VLAN ID.

– Each interface VLAN has the following attributes:

•Admin state

•Operational state

•MAC address

•IP address and mask

•MTU

•Description•Set of counters

93

Configure a Router Port Interface

94

Q&A

– Question ?

– Break 10 m

95

Basic Troubleshooting


What happens if the IPL link goes down?

97

Link Aggregation

Data VLAN 300

ILO


SN2410M

Cisco MGMG switch

VLAN 400

mgmt0

iLO

Qumulo

Servers 1-6

MAGP 1:Interface vlan: 300Admin state : EnabledState : MasterVirtual IP : 10.0.1.254Virtual MAC : 00:00:5E:00:01:01

MAGP 2:Interface vlan: 400Admin state : EnabledState : MasterVirtual IP : 10.0.102.1Virtual MAC : 00:00:5E:00:01:02 10.0.99.61 ‐ 63 /24

inter-peer link (IPL) Management Plane (Active/Standby)

SN2410Mmgmt0

1. Split-brain2. Only the master switch will pass traffic.

What happens if no IP communication between the MGMT ports

98

Link Aggregation

Data VLAN 300

ILO


SN2410M

Cisco MGMG switch

VLAN 400

mgmt0

iLO

Qumulo

Servers 1-6

MAGP 1:Interface vlan: 300Admin state : EnabledState : MasterVirtual IP : 10.0.1.254Virtual MAC : 00:00:5E:00:01:01

MAGP 2:Interface vlan: 400Admin state : EnabledState : MasterVirtual IP : 10.0.102.1Virtual MAC : 00:00:5E:00:01:02 10.0.99.61 ‐ 63 /24

Management Plane (Active/Standby)

SN2410Mmgmt0

1. CLI prompt is displayed: [:unknown]#2. It Split-brain when IPL down

Verify interface MLAG port channel

99

Link Aggregation

Data VLAN 300

ILO


SN2410M

Cisco MGMG switch

VLAN 400

mgmt0

iLO

Qumulo

Servers 1-6

10.0.99.61 ‐ 63 /24

SN2410Mmgmt0

#show interface mlag-port-channel sum

mlag-port-channel

Q&A

– Question ?

100

Thank [email protected]

hpe mellanox switch training - home | iotlabs...pptv network diagram 3 link aggregation data vlan...

Documents