arria 10 low latency 10g ethernet mac and 1g/2.5g/5g/10g ...€¦ · the system side of aquantia...

©2017 Intel Corporation. All rights reserved. ALTERA, ARRIA, CYCLONE, HARDCOPY, INTEL, MAX, MEGACORE, NIOS, QUARTUS and STRATIX words and logos are trademarks of Intel Corporation and registered in the U.S. Patent and Trademark Office and in other countries. All other words and logos identified as trademarks or service marks are the property of their respective holders as described at www.altera.com/common/legal.html. Intel warrants performance of its semiconductor products to current specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any products and services at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.

Date: Aug 2017

Revision: 1.0

Arria 10 Low Latency 10G Ethernet MAC and

1G/2.5G/5G/10G Multi-Rate Ethernet PHY

Interoperability Hardware Demonstration with

Aquantia PHY Reference Design

2

Table of Contents Introduction .................................................................................................................................................. 3

Functional Flow For NBASE-T Ethernet Interoperability System .................................................................. 4

Design Components ...................................................................................................................................... 6

Hardware and Software Requirements ........................................................................................................ 8

Hardware Testing .......................................................................................................................................... 9

Speed Change Test .................................................................................................................................. 15

Aquantia PHY Loopback mode ................................................................................................................ 22

Reference Design Debug Features .......................................................................................................... 25

3

Introduction

This application note describes a design that demonstrates the interoperability between Intel Low

Latency 10G Ethernet IP solution and Aquantia PHY device. It uses Intel’s Arria 10 Low Latency Ethernet

10G Media Access Controller (MAC) and 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP cores in NBASE-T

mode on Arria 10 GX Transceiver Signal Integrity Development Kit together with a Aquantia Evaluation

Board which contains the AQR105 Ethernet PHY device.

This design offers the following features:

• Auto negotiation test between Intel Arria 10 Field Programmable Gate Array (FPGA) and

Aquantia AQR105.

• Transmission and reception of packets between Arria 10 FPGA and Aquantia AQR105.

• Sequential random bursts tests. You can configure the number of packets, payload-data type,

and payload size for each burst.

• Packet statistics for traffic generator, monitor, MAC transmitter (TX) and MAC receiver (RX).

• Packet classification for different frame sizes transmitted and received by the MAC.

• Throughput for the traffic received by the traffic monitor.

• System Console user interface. This tool command language (Tcl)-based user interface allows

you to dynamically configure and monitor any registers in the reference design.

4

Functional Flow For NBASE-T Ethernet Interoperability System

The following figure illustrates the overview of the design’s system architecture. In this system, the

Arria 10 FPGA will interact with the Aquantia AQR105.

Figure 1. System Architecture Overview

Arria 10 Transceiver Signal Integrity Development Kit

Arria 10 GX FPGA

NBASE-T Ethernet Subsystem (LL10GMAC

+ MGE PHY)

SFP+

10GbE SFP+ Direct Attach

Copper(System Side)

Aquantia Ethernet PHY Evaluation Board

Aquantia AQR105 PHY

Ethernet Cat6a cable (Line Side)

Aquantia Ethernet PHY Evaluation Board

Aquantia AQR105 PHY


Arria 10 GX FPGA


+ MGE PHY)SFP+

10GbE SFP+ Direct Attach

Copper(System Side)

JTAG JTAG

PC and System Console

Subsystem

Aquantia GUI (PC)

1. Based on the Figure 1 setup, it contains 2 Ethernet stations (which each of it consist of the Arria

10 FPGA development kit and Aquantia AQR105 evaluation board). Both stations connected

with each other using the category (CAT) 6A cable.

2. The design used to load into both Intel Arria 10 Transceiver Signal Integrity (SI) development kit

will be based on the Intel Arria 10 10G Universal Serial Media Independent Interface (USXGMII)

Ethernet Design Example. Refer to Intel Arria 10 Low Latency Ethernet 10G MAC Design

Example User Guide, section 10G USXGMII Ethernet Design Example for details.

3. The PC running on a Linux or a Window operating system, preinstalled the Quartus Prime will be

used to configure and control the FPGA. Hardware testing on FPGA side will be demonstrated

using the Tcl-based System Console from Quartus Prime software. For Aquantia PHY side, the

Aquantia GUI tool will be used to provide the access to the Aquantia PHY’s Management Data

Input/Output (MDIO) registers to control and monitor the link performance.

5

4. Connection between the transceiver GXB side of FPGA to the Aquantia AQR105 will be using the

small form-factor pluggable transceiver (SFP+) interface. A 10GbE SFP+ direct attach copper

cable is used to connect between the SFP+ cage on Arria 10 Transceiver SI development kit to

the system side of Aquantia AQR105 evaluation board.

5. This design will include various hardware testing, such as auto negotiation test and basic traffic

check test. (The test setup details will be discussed on the following section)

Related Links

Intel Arria 10 Low Latency Ethernet 10G MAC Design Example User Guide

https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/ug/ug-20016.pdf

Section 1 Quick Start Guide on page 5

Describes on how to generate the design example from Quartus Prime IP Parameter Editor.

Section 10G USXGMII Ethernet Design Example on page 71

Describes the design example features, function description, architecture and steps to perform the

hardware testing.


6

Design Components

The reference design consists of four main subsystems: Arria 10 FPGA NBASE-T Ethernet Subsystem,

Aquantia ARQ105, Aquantia Graphical User Interface (GUI) and System Console.

Table 1. Intel FPGA Subsystem Design Components

This table lists the components of the FPGA design’s subsystems.

Component Description

Low Latency Ethernet 10G MAC IP core This IP core handles the flow of data through

the Multi-rate Ethernet PHY IP core.

• On transmit path, the MAC accepts client

frames and constructs Ethernet frames before

forwarding them to the PHY layer.

•On the receive path, the MAC accepts

Ethernet frames through the PHY layer,

performs checks and removes the relevant

fields before forwarding the frames to the

client. For this design, the MAC uses the

memory-based statistics counters.

1G/2.5G/5G/10G Multi-rate Ethernet PHY IP

core

The PHY IP core will be using the

1G/2.5G/5G/10G (USXGMII) variant mode.

Transceiver Reset Controller The Intel Transceiver PHY Reset Controller IP

core. It helps to reset the transceiver.

ATX PLL Generate a TX serial clock for the Arria 10

Transceiver.

IO PLL Generates clocks for all design components.

Traffic Controller The traffic controller consists of:

• Traffic generator—injects client packet bursts

into the MAC TX core.

7

• Traffic monitor—receives packet bursts from

the MAC RX core.

The traffic controller connects to the Avalon-ST

single-clock FIFO in the Ethernet subsystem

through the Avalon Streaming (Avalon-ST)

interface.

Ethernet Packet Generator This module consists of Avalon Memory-

Mapped (Avalon-MM) registers, Ethernet

packet generation block, CRC generator, and

shift register.

Ethernet Packet Monitor This module verifies the payload of received

packets and collects information from the

statistics counters. This consists of Avalon-MM

registers and CRC checkers

JTAG to Avalon Master Bridge This IP core provides a connection between the

System Console and Qsys system through the

physical interfaces. The System Console

initiates Avalon-MM transactions by sending

encoded streams of bytes through the bridge’s

physical interfaces.

8

Hardware and Software Requirements

Intel uses the following hardware and software to test the design.

Hardware

• Intel Arria 10 GX Transceiver Signal Integrity (SI) development kit (Qty: 2)

• Aquantia ARQ105 evaluation board (Qty: 2)

• 10GbE SFP+ direct attach copper cable (Qty: 2)

• Ethernet CAT 6A cable

Software

• Quartus® Prime Standard version 17.0 (for hardware testing)

• Future Technology Device International (FTDI) driver*

• Aquantia USXGMII firmware*

• Aquantia PHY GUI*

*For Aquantia related firmware and software, please register on Aquantia Customer Portal on the link

below:

https://portal.aquantia.com/customer-portal

For a new user, please register on the website and create a new account. After the account is approved

by Aquantia, user may login into the portal and obtain all the required firmware and software. The

firmware and software will be update from time to time. Please consult Aquantia for the latest version

of firmware and software available. (email: [email protected])


mailto:[email protected][email protected]

9

Hardware Testing

Follow the steps below to perform the hardware test:

1. Install the FTDI driver from the link below. (It is required for the Aquantia ARQ105 evaluation

board micro-USB access port. Driver must be installed for the evaluation board’s connection

over the USB port. Connect the micro-USB port to the PC to install the FTDI driver.)

2. Next, connect the Aquantia board to PC. Program the Aquantia board with the USXGMII

firmware provided by Aquantia. (The firmware (cld file) and software can be downloaded from

the Aquantia Customer Portal. Detail steps can refer to section above. All the files were zipped

in the file named as AQGUI_USB_ShuttleTools.zip)

3. Extract the AQGUI_USB_ShuttleTools zip file and store the firmware together into a similar

directory in PC. Open a command prompt and change the directory to where the Aquantia

related files located. Next, type the command below in command prompt to flash the firmware

into the Aquantia AQR105 evaluation board.

Command: flashburn.exe <target_board_part_number> 0 <firmware_name>.cld

4. After the firmware is successfully flash into the board, open the Aquantia GUI

(AQGUI_USB_Shuttle.exe) and choose the devices option to see the PHY status. (The links are

expected to not link up for both PHYs as our FPGA have not go through any bring up process

yet.)

5. The Aquantia GUI is only applicable for user set to NBASE-T mode. It is not compatible when

user used the IEEE802.3bz mode. By default, user will be using the NBASE-T mode. NBASE‐T is a

new Ethernet signaling method that has been adopted as the basis of IEEE 802.3bz. Newer PHY

firmware may have IEEE802.3bz mode enabled. You can manually set the IEEE802.3bz registers

like the speed rate advertisements under the Register Access box at the bottom of the GUI.

10

Figure 2. Aquantia GUI

6. Generate the Arria 10 10G USXGMII Ethernet design example from Quartus Prime 17.0 Standard

edition IP Parameter Editor. (Refer to section 1 on Intel Arria 10 Low Latency Ethernet 10G MAC

Design Example User Guide for the detail steps on how to perform the generation.)

7. Open the Clock Control application, which is the part of development kit. You may download it

from the Arria 10 GX Transceiver SI development kit link below. From the directory

arria10GX_10ax115sf45_si_v15.1\examples\board_test_system \ClockController.exe, open

the tool and set Y5 to 644.53125 MHz and Y6 to 125 MHz. (Remember to perform this changes

to both dev kits. If you connected to 2 dev kits on one single PC, a pop up will prompt up for

choosing which path you would to modify after you open the ClockController.exe)

8. Open the project file (altera_eth_top.qpf) of Arria 10 10G USXGMII Ethernet design example in

the Quartus Prime software.

11

9. Click Processing ➤ Start Compilation to compile the design.

10. After the compilation completed, configure the FPGA using the generated configuration file

(altera_eth_top.sof).

11. Modification needed to be perform on the Tcl file for hwtesting. First, look for the

system_console folder from directory LL10G_10G_USXGMII/hwtesting/system_console. Then,

duplicate another version of this folder. For example, you will get 2 folders, rename it to

system_console_pod13_A and system_console_pod13_B.

12. Since one PC is used to connect with 2 Arria 10 GX SI development kits, both kits will be using

different paths for the System Console command. Maintain the system_console_pod13_A with

no changes is required. For system_console_pod13_B, open the basic.tcl from directory

system_console_pod13_B/basic and change the following:

Table 1. System Console basic.tcl file command name

From To

set port_id [lindex [get_service_paths master]

0];

set port_id [lindex [get_service_paths master]

1];

13. In the Quartus Prime software, select Tools ➤ System Debugging Tools ➤ System Console to

launch the system console.

14. Change the working directory to LL10G_10G_USXGMII /hwtesting/ system_console_pod13_A.

15. Initialize the design command list by running this command: source main.tcl

16. Run the following command in the system console to initialize the channel and enable the auto-

negotiation test:

SET_CHANNEL_BASE_ADDR 1

SET_PHY_USXGMII_AN

17. Perform the similar command from steps 13 to 15 for different directory, changed to

LL10G_10G_USXGMII /hwtesting/ system_console_pod13_B.

18. Next, we need to turn on the auto negotiation mode for the Aquantia PHY. Using the Aquantia

GUI, choose the device and go to Utilities > Register Guide and write register 4.C441 with value

0x8808 to enable the auto negotiation on the PHY. After you successfully bring up, you will

12

observe from the figure below that the link is up. (We need to enable for both Aquantia

evaluation board.)

Figure 3. Aquantia GUI (Auto negotiation enabled)

19. After both links are up on both Aquantia boards, we will continue the traffic test using the

system console.

20. Back to the system console, change the working directory to LL10G_10G_USXGMII /hwtesting/

system_console_pod13_A. Initialize the design command list by running this command: source

main.tcl

21. Run the following command in the system console to start the test. (The test must run at

channel 1 only as the design example is assigned the channel 1 of transceiver to the SFP+

interface.)

TEST_EXT_LB <channel> <speed> <burst_size>

Example: TEST_EXT_LB 1 10G 80000000

22. For a successfully run, you will observe the packet monitor results such as below for both FPGAs.

When the test completed on one path, the following diagrams show that the packet monitor

13

block received the same number of packets generated bt traffic generator without error,

together with the TX and RX statistics counters details.

Figure 4. Sample Output – Packet Monitor

14



LL10G_10G_USXGMII /hwtesting/ system_console_pod13_B. Similar with previous results, you

need to observed no error packet observed from the packet monitor.

15

Speed Change Test

This test is basically covered the steps to perform the auto negotiation test between the Arria 10 FPGA

and Aquantia ARQ105 PHY. Auto negotiation is an Ethernet procedure protocol by which two devices

connecting with each other and choose to use the common transmission parameter. In the process, one

of the connected device will first share their capabilities and then choose the highest performance

transmission mode they both support. In our test case, we will use the speed as the transmission

parameter. The following steps will guide user on how to perform the speed change test. User need to

make sure the testing the design steps are completed before continue on the steps below.

1. After completed the Testing the design section on basic bring up on both Arria 10 SI kit and

Aquantia evaluation board, next user can continue with the more comprehensive speed

switching test. First, user may perform the first condition where speed change done on Aquantia

PHY side, Arria 10 FPGA will negotiate to the correct speed.

2. Next, open the Aquantia GUI, disable the unwanted speed and press the restart Autonegotiation

button. For example, if user want to enable 5G speed, need to untick the 10Gb speed option on

the auto negotiation section from the GUI. By performing this changes, the Aquantia PHY will

connect to 5G as this is the highest speed among others remaining. (It is expected based on the

auto negotiation behavior.) Perform the same changes on both Aquantia PHYs, using 2

separated GUI. Figure below shows the option of the speed selection.

16

Figure 6. Aquantia PHY GUI – Speed selections

3. The following table show all the combination that user can set to perform the speed change

test.

Table 2. Speed Mode

Aquantia PHY (Speed mode) Auto-negotiated speed

1G/2.5G/5G/10G 10G

1G/2.5G/5G 5G

1G/2.5G 2.5G

1G 1G

4. Next, user need to continue the configuration on FPGA side. In the Quartus Prime software,

select Tools ➤ System Debugging Tools ➤ System Console to launch the system console.

5. Change the working directory to LL10G_10G_USXGMII /hwtesting/ system_console_pod13_A.

6. Initialize the design command list by running this command: source main.tcl

17

7. After the speed is set on the Aquantia PHY, user can restart the auto negotiation (AN) on Arria

10 FPGA (if the auto negotiation is enabled previously) on system console. Type the command

below on system console to restart the AN process on FPGA.

Command: SET_CHANNEL_BASE_ADDR 1

Command: SETPHY_USXGMII_AN_RESTART

8. If user do not enable the auto negotiation speed previous, run the following command in the

system console to initialize the channel and enable the auto-negotiation test:

Command: SET_CHANNEL_BASE_ADDR 1

Command: SET_PHY_USXGMII_AN

9. Next, key in the command below on system console to check the Multi-rate Ethernet PHY IP

core register settings.

Command: CHK_PHY_STATUS

10. Figure below shows the example result for the PHY status reading. Those PHY register

definitions are available on Arria 10 Transceiver PHY user guide.

Figure 7. PHY status register result

Table below shows the description on all the USXGMII register settings:

18

Table 3. USXGMII Registers detail

USXGMII Control Register Bit [0]

• 0: 10GBASE-R mode

• 1: USXGMII mode

Bit [1]

• 0: Disable USXGMII Auto-Negotiation

• 1: Enable USXGMII Auto-Negotiation

USXGMII Status Register Bit [2]

• 0: Link is up

• 1: Link is down

Bit [5]

• 1: Auto-negotiation process is

completed.

USXGMII Partner Ability Register Bit [11:9] Speed

• 3’b010 : 1G

• 3’b011: 10G

• 3’b100: 2.5G

• 3’b101: 5G

Bit [15] Link status

• 0: Link down

• 1:Link up

All the register mentioned above are important for user to track the status of auto negotiation

on FPGA. For full PHY register definitions, please refer to Arria 10 Transceiver PHY user guide.

11. Next, user need to repeat the steps 4 to 10 by changing to another FPGA on different path.

(The directory is LL10G_10G_USXGMII /hwtesting/ system_console_pod13_B.)

12. Make sure both FPGAs can initiate to the respective speed set by the Aquantia PHY. Check

for the cable connectivity if the link is not able to bring up. User may follow the steps on the

section below, description on how to enable the loopback mode on Aquantia PHY to

perform any debugging test.

19

13. After the speed switch completed for the system, we will continue the traffic test using the

system console. Back to the system console, change the working directory to

LL10G_10G_USXGMII /hwtesting/ system_console_pod13_A. Initialize the design command list

by running this command: source main.tcl

14. Run the following command in the system console to start the test. (The test must run at

channel 1 only as the design example is assigned the channel 1 of transceiver to the SFP+

interface.)

TEST_EXT_LB <channel> <speed> <burst_size>

Example: TEST_EXT_LB 1 10G 80000000

15. For a successfully run, you will observe the packet monitor results such as below for both FPGAs.

When the test completed on one path, the following diagrams show that the packet monitor

block received the same number of packets generated bt traffic generator without error,

together with the TX and RX statistics counters details.

20


21



LL10G_10G_USXGMII /hwtesting/ system_console_pod13_B. Similar with previous results, you

need to observed no error packet observed from the packet monitor.

22

Aquantia PHY Loopback mode

Aquantia PHYs support several test modes in all the supported line-rates providing a flexible and

effective debug capability to narrow down the point of failure within the PHY and to eliminate any

possible cause of errors. This debug mode is useful for user when implementing Intel FPGA and Aquantia

PHY with external devices, such as Ethernet switch or router.

There are two main loopback modes supported by Aquantia PHYs:

1. Line Side Loopback mode

• Packets received from the line-side interface are sent back to the link-side interface.

• There are two type of line side loopback supported, which is the shallow and deep type.

• Figure 5 below shows this loopback mode is named as Network - System Loopback

(Deep system facing loopback).

2. System Side Loopback mode

• Packets received from the system-side interface and sent back to the system-side

interface.

• This mode is not tested in this design as not applicable to use as the debug mode.

When the loopback mode is enabled, auto negotiation function will be disabled. User need to manual

set the loopback rate. All the loopback enable and set the loopback rate can be done by using the

Aquantia PHY register map. When the loopback mode enabled, user must set the loopback rate in order

to make the link up. User need to turn on the loopback mode for the Aquantia PHY by using the

Aquantia GUI, choose the device and go to Utilities > Register Guide and write register 4.C444. Table

below shows the register values for different loopback types and rates.

23

Table 4. Aquantia PHY Register 4.C444 Loopback Type

Register 4.C444.[F:B] Loopback Type

0x00 No loopback

0x01 System - System Loopback (Shallow system facing loopback)

0x03 System - Network Loopback (Deep network facing loopback)

0x09 Network - System Loopback (Deep system facing loopback)

0x0B Network - Network Loopback (Shallow network facing loopback)

Table 5. Aquantia PHY Register 4.C444 Loopback Rate

Register 4.C444.[2:0] Loopback Rate

0x0 Reserved

0x1 100Mbps

0x2 1Gbps

0x3 10Gbps

0x4 2.5Gbps

0x5 5Gbps

24

Figure 5. Aquantia supported loopback paths

PCS PMAUSXGMII

System I/F

Line

System Interface – System Loopback (PCS loopback)

Network Interface – System Loopback (PMA loopback)

System Interface – Network Loopback (PHY-XS loopback)

Network Interface – Network Loopback

25

Reference Design Debug Features

The debug strategy below helps user to make a self-debug if you encounter system bring up issue. The

recommended steps below help make a high level isolation on which device cause the link down, either

the FPGA system or the Aquantia PHY side.

Refer to steps below on how to perform the FPGA only test:

1. Change the test setup with the reference to Figure 7.

Figure 6. NBASE-T Sub system (FPGA loopback mode, without Aquantia PHY)


Arria 10 GX FPGA


+ MGE PHY)

SFP+


Arria 10 GX FPGA


+ MGE PHY)SFP+

JTAG JTAG

PC and System Console

Subsystem

SFP+ loopback adapter module SFP+ loopback adapter

module

2. Next remove the two Aquantia evaluation boards, and connect the SFP+ cage of Arria 10

Transceiver SI kit with a SFP+ loopback adapter module which is used for 10GbE Ethernet

compliance testing and copper/optical cable emulation.

3. For user that have not completed the steps to bring up the test from testing the design section,

may refer to steps 5 – 14 to bring up both FPGAs, and continue with step 20 -22 to run the

traffic test on both FPGA kits and the traffic results from both TX and RX statistic counters.

26

Related Links

FTDI Driver

http://www.ftdichip.com/FTDrivers.htm

Aquantia Customer Portal


Intel Arria 10 Low Latency Ethernet 10G MAC Design Example User Guide


Section 1 Quick Start Guide on page 5

Describes on how to generate the design example from Quartus IP Parameter Editor.

Arria 10 GX Transceiver SI Development Kit Downloadable Content

https://www.altera.com/products/boards_and_kits/dev-kits/altera/kit-a10-gx-si.html

Download the Arria 10 GX Transceiver Signal Integrity Installation Package (Production Edition)

Arria 10 Transceiver PHY User Guide

Section 2.6.5 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP Core page 198

Contain the PHY registers definition.

http://www.ftdichip.com/FTDrivers.htm



https://www.altera.com/products/boards_and_kits/dev-kits/altera/kit-a10-gx-si.html

https://www.altera.com/content/dam/altera-www/global/en_US/support/boards-kits/arria10/signal-integrity-dev-kit/arria10GX_10ax115sf45_si_v15.1.zip

arria 10 low latency 10g ethernet mac and 1g/2.5g/5g/10g ...€¦ · the system side of aquantia...

Documents