Link AggregationOver View of the IEEE 802.3ad-2000

(clause 43)

Authors: Anthony Mwingira, Jon Sterritt

Bridge Functions Consortium

IntroductionLink Aggregation (LA): The combination of

multiple physical links to function as a single logical link.

Link Aggregation Group (LAG): A group of physical links that make up a single aggregation. Each physical port may be a member of a single LAG.

Link Aggregation Control Protocol (LACP): The protocol specified by IEEE 802.3.ad-2000 to outline standardized Link Aggregation.

Possible Problems and Issues By combining multiple physical links to form a

single logical link two notable issues could occur: Frame duplication, “flooding” Frame disordering

Both of these issues are addressed by LACP.

Goals and Objectives of LA Increased bandwidth – multiple links combined into one

logical link Linearly incremental bandwidth – increase in unit

multiples Increased availability – failure of a single link within aggr.

Need not cause MAC client failure Load Sharing – MAC Client traffic may be distributed

across links Auto Configuration – in even of changes in physical

connection Rapid configuration and reconfiguration – also in the

event of changes in physical connection

Deterministic behavior – independent of algorithm chosen( implementer can choose internal algorithm). The configuration can be made to resolve deterministically i.e. resulting aggr. Can be made independent of the order in which events occur, and be completely determined by the capabilities of the individual links and their physical connectivity.

Low risk of duplication or mis-ordering – during link reconfiguration or steady-state operation there is a high probability that frames are neither duplicated or mis-ordered.

Backwards compatibility with aggregation-unaware devices - Link that can not take part in an aggregation operate as normal, individual links.

Multipoint Aggregations – does not support aggregations among more than two systems; multiple aggregations are allowed, but 3 systems cannot share a single Link Aggregation Group.

Limitations:MACs – supported only on links using

the IEEE 802.3 MACHalf-duplex operation – supported

only on point-to-point links with MACs operating in full duplex mode.

Multiple data rates – All Links in a Link Aggregation Group operate at the same data rate (e.g. 10 Mb/s, 100 Mb/s or 1000 Mb/s )

LAG Operation Frame Distribution - Responsible for taking frames from MAC Client and

submitting them for transmission on appropriate ports. Frame Collection – Responsible for passing frames received from the various

ports to the MAC Client -> also includes Marker Responder ( Marker protocol – allows the distribution function of an Actor’s Link Aggregation sub-layer to request the transmission of a Marker PDU on a given link)

Aggregator Parser/ Multiplexers. When transmitting these blocks pass frame requests to the appropriate port. When receiving, distinguish frames and pass them to appropriate entity.

Aggregator – This is the combination of frame Distribution and Collection, along with the aggregator Parser/ Multiplexers.

Aggregation Control – Responsible for configuration and control of the Link Aggregation.

Control Parser/Multiplexers – On transmit -> pass frame transmission requests from the Aggregator and Control entities to the appropriate port. On receive, these blocks distinguish Link Aggregation Control PDUs from other frames, passing LACPDUs to the appropriate sub-layer entity, and all other frames to the Aggregator.

Principles of Link Aggregation Note: Link Aggregation allows a MAC Client to treat a set of

one or more ports as if it were a single port.

A Mac Client communicates with a set of ports through an Aggregator. An Aggregator binds to one or more ports within a System.

It is the responsibility of an Aggregator to distribute frame transmissions from the MAC Client to the various ports, and to collect received frames from the ports and pass them to the MAC Client transparently.

A given port will bind to (at most) a single Aggregator at any time. A MAC Client is served by a single Aggregator at a time.

The binding of ports to Aggregators within a System is managed by the Link Aggregation Control function for that System.

a unique, globally administered MAC address.

Binding may be under manual control through direct manipulation of the state variables of Link Aggregation (e.g. Keys) by net manager. Automatic determination, configuration, binding, and monitoring my occur through the use of Link Aggregation Control Protocol (LACP)

Frame ordering must be maintained. The distributor ensures that all frames of a given conversation are past to a single port. They are to be passed in order they were received from that port.

Conversations may be moved among ports within aggregation, both for load balancing and to maintain availability in the event of a link failures.

Each port is assigned a unique , globally administered MAC address. ( Prospective of a MAC Client)

Each Aggregator is assigned a unique, globally administered MAC address.

Link Aggregation Control Link Aggregation Control configures and

controls the Link Aggregation sub layer using static information local to the function and dynamic information exchanged by means of the Link Aggregation Control Protocol.

For Each aggregatable port in the System: Maintains Configuration information to

control aggregation

Exchange configuration information with other with other Systems to allocate the link to a Link Aggregation Group. NOTE a given link is allocated to, at most, one Link Aggregation Group at a time.

Attaches the port to the Aggregator used by the Link Aggregator Group, and detaches the port from the Aggregator when it is no longer used by the Group

Uses information from the Partner Systems’ Link Aggregation Control entity to enable or disable the Aggregator or Distributor

Checking that candidate links are actually be aggregated.

Controlling link addition to Aggr. Group and the creation of a Group if necessary

Monitoring the status of aggregated links Removing a Link from a Link Aggr. Group if

its membership is no longer valid, and removing the group if it no longer has any member links.

Test Suite – General How To See Cisco Document, most devices work just

like Cisco with different variations, all the following could port a port into LAG1 in active mode depending on the device. “channel-group 1 mode active” (Cisco) “channel-group 1 mode auto” (some Cisco) “port-channel” 1 mode auto “LACP mode auto”

“LACP key 1” (Foundry uses keys, no LAG’s)

How To Configure LAG’SDifferent Strategies by Vendor#1 Cisco - go into the interface prompt and put that

interface into a LAG, i.e.,“channel-group”#2 Dell - command line, almost identical to Cisco, Web

Interface the ports are configured from the LAG menu

#3 HP, Nortel - LAG’s configured as “trunks” that run LACP as a protocol.

#4 3Com – From menu driven command line ports must be activated for LACP (port menu) and then added to an activated LAG (Link Agg. Menu).

#5 Other “odd” implementations require setting keys, most devices require ports assigned to a LAG, but some require setting the Key for each port, the result is equivalent.

More About Configuration #1 Configure LAG’s, some tests require multiple LAG’s

Complete Cabling/Setup

Ensure that for the device you are working on there is not a way to configure the LAG as an interface – i.e., sometimes you can configure the group as if it was a port – allow speed, enable/disable, and other configurations to be executed on the entire group of physical links that make up the LAG

See the Dell web Interface: Ports->LAG Configuration See the Cisco Interface: int port-channel <PORT_CHAN>

Trouble-Shooting/Testing Continued Line Taps: Very useful when diagnosing

issues with LACP Verify that LACP PDU’s are being transmitted Verify that both devices are correctly transmitting

traffic Usually if inter-operability problems as suspect it

will be determined that one device is not transmitting or receiving frames on it’s aggregated links

Trouble-Shooting/TestingWhen a port is enabled for active LACP, PDU’smust be transmitted from that port:

MAC 01:80:C2:00:00:02 Type/Length 8809

Also, check the status of the LAG’s on both devices; LAG statusshould be “bundl” (Cisco) “oper”, “operational”, “active”, orequivalent and NOT “indep” (Cisco), “down”, “dis”, etc..

With STP DISABLED traffic should be passed over a multi-portLAG without flooding or duplication, with multiple LAG’s floodingshould occur unless there is a protocol in place to prevent loops.

Note: Some devices will disable ports in the event of flooding.

Most LA testing is executed with STP disabled for this reason – ifLACP fails we want to note flooding.

Specific Device Configuration Cisco – See My Cisco Document (word doc) The following is basic Dell CLI config.Note: Web Interface Config is very easy, simply use the web interface

LAG menu and specify a LAG and “L” for LACP in the table.

console# confconsole(config)# int eth g1console(config-if)# channel-group 1 mode autoconsole(config-if)# exitconsole(config)# int eth g2console(config-if)# channel-group 2 mode auto

console# show lacp port 1

Console#show lacp port 1Port-Channel ch1 Port Type Gigabit Ethernet Attached Lag id: Actor System Priority:1 MAC Address: 00:30:ab:cd:66:55 Admin Key: 25 Oper Key: 25 Partner System Priority:0 MAC Address: 00:00:00:00:00:00 Oper Key: 0console# show lacp port 2Port-Channel ch2 Port Type Gigabit Ethernet Attached Lag id: Actor System Priority:1 MAC Address: 00:30:ab:cd:66:55 Admin Key: 26 Oper Key: 26 Partner System Priority:0 MAC Address: 00:00:00:00:00:00 Oper Key: 0

Produces the following LAG configuration.

3Com 4400 (RSTP) Config

Login: adminPassword:

Menu options: --------------3Com SuperStack 3 Switch 4400--------------- bridge - Administer bridge-wide parameters feature - Administer system features gettingStarted - Basic device configuration logout - Logout of the Command Line Interface physicalInterface - Administer physical interfaces protocol - Administer protocols security - Administer security system - Administer system-level functions trafficManagement - Administer traffic management

Type ? for help----------------------------------------- (1)---------------------------Select menu option: bridge

Menu options: --------------3Com SuperStack 3 Switch 4400--------------- addressDatabase - Administer bridge addresses broadcastStormCont - Enable/disable broadcast storm control linkAggregation - Administer aggregated links multicastFilter - Administer multicast filtering port - Administer bridge ports resilientLinks - Administer resilient links spanningTree - Administer spanning tree summary - Display summary information vlan - Administer VLANs

Type "quit" to return to the previous menu or ? for help----------------------------------------- (1)---------------------------Select menu option (bridge): linkA

Menu options: --------------3Com SuperStack 3 Switch 4400--------------- detail - Display detailed information modify - Modify aggregated links summary - Display summary information

Type "quit" to return to the previous menu or ? for help----------------------------------------- (1)---------------------------

Select menu option (bridge/linkAggregation): mod

Menu options: --------------3Com SuperStack 3 Switch 4400--------------- addPort - Add a port to an aggregated link linkState - Enable/disable an aggregated link partnerID - Set the partner ID for an aggregated link removePort - Remove a port from an aggregated link

Type "quit" to return to the previous menu or ? for help----------------------------------------- (1)---------------------------Select menu option (bridge/linkAggregation/modify): addPortSelect aggregated link index (1-4): 1Select ports (unit:port...,?): 1:1

Select menu option (bridge/linkAggregation/modify): q

Menu options: --------------3Com SuperStack 3 Switch 4400--------------- addressDatabase - Administer bridge addresses broadcastStormCont - Enable/disable broadcast storm control linkAggregation - Administer aggregated links multicastFilter - Administer multicast filtering port - Administer bridge ports resilientLinks - Administer resilient links spanningTree - Administer spanning tree summary - Display summary information vlan - Administer VLANs

Type "quit" to return to the previous menu or ? for help----------------------------------------- (1)---------------------------Select menu option (bridge): port

Menu options: --------------3Com SuperStack 3 Switch 4400--------------- defaultPriority - Sets the default 802.1D priority for untagged frames detail - Display detailed information lacpState - Enable/disable Link Aggregation Control Protocol stpCost - Set the Spanning Tree path cost parameter stpFastStart - Enable/disable Spanning Tree fast start summary - Display summary information

Type "quit" to return to the previous menu or ? for help----------------------------------------- (1)---------------------------Select menu option (bridge/port): lacpSSelect ports (unit:port...,?): 1:1Enter new value (enable,disable)[enable]: en

Select menu option (bridge/port):

HP Configuration

HP ProCurve Switch 4172GL 8-Jan-1990 0:35:50==========================- CONSOLE - MANAGER MODE -==========================18 Main Menu

1. Status and Counters... 2. Switch Configuration... 3. Console Passwords... 4. Event Log 5. Command Line (CLI) 6. Reboot Switch 7. Download OS 8. Run Setup 9. Stacking... 0. Logout

Displays the event log.To select menu item, press item number, or highlight item and press <Enter>.

Enter 2 to get to the following menu:

HP ProCurve Switch 4172GL 8-Jan-1990 0:38:==========================- CONSOLE - MANAGER MODE -========================== Switch Configuration Menu

1. System Information 2. Port/Trunk Settings 3. Network Monitoring Port 4. Spanning Tree Operation 5. IP Configuration 6. SNMP Community Names 7. IP Authorized Managers 8. VLAN Menu... 0. Return to Main Menu...

Configures system-level information including system identification.To select menu item, press item number, or highlight item and press <Enter>.

Enter 2 to get to the following menu:

HP ProCurve Switch 4172GL 8-Jan-1990 0:42:13==========================- CONSOLE - MANAGER MODE -==========================18 Switch Configuration - Port/Trunk Settings

Port Type Enabled Mode Flow Ctrl Group Type ---- --------- + ------- ------------ --------- ----- ----- A1 100FX | Yes 100FDx Disable A2 | Yes Disable A3 | Yes Disable B1 10/100TX | Yes Auto Disable B2 10/100TX | Yes Auto Disable B3 10/100TX | Yes Auto Disable Trk1 LACP B4 10/100TX | Yes Auto Disable Trk1 LACP B5 10/100TX | Yes Auto Disable Trk2 LACP B6 10/100TX | Yes Auto Disable Trk2 LACP B7 10/100TX | Yes Auto Disable B8 10/100TX | Yes Auto Disable B9 10/100TX | Yes Auto Disable

Actions-> Cancel Edit Save Help

Cancel changes and return to previous screen.Use arrow keys to change action selection and <Enter> to execute action.

Use the arrow keys and space bar to configure.

Notice Two LAG’s of Two ports each.