Building Networks with Open,

1st A i N tFPGA D l ’ W k h

Reconfigurable Hardware1st Asia NetFPGA Developer’s Workshop

Monday, June 14, 2010

John W. LockwoodCEO & Lead Consultant, Algo-Logic Systems

With input from: NetFPGA Group,Stanford University

1 Algo-Logic

NetFPGA’s Defining Characteristics• Line-Rate

– Processes back-to-back packets • Without dropping packetsWithout dropping packets • At full rate of Gigabit Ethernet Links

– Operating on packet headers • For switching, routing, and firewall rules

– And packet payloads• For content processing and intrusion prevention

• Open-source Hardware– Similar to open-source software

• Full source code available • BSD-Style License

– But harder, because • Hardware modules must meeting timing• Verilog & VHDL Components have more complex interfaces

2 Algo-Logic

• Verilog & VHDL Components have more complex interfaces • Hardware designers need high confidence in specification of modules

NetFPGA = Networked FPGAA line-rate, flexible, open networking platform for teaching and research

3 Algo-Logic

NetFPGA Reference PlatformN tFPGA

CPU Memory

NetFPGA

=

Softwarerunning on a

1GE

PCI PC with NetFPGAstandard PC

+

FPGA1GE

1GE

1GE

A hardware acceleratorbuilt with Field

Memory 1GENetFPGA Board

built with Field Programmable Gate Arraydriving Gigabit

4 Algo-Logic

driving Gigabit network links

NetFPGA System

Browser& VideoMonitor

SoftwareCADT l

Web &Video

User Space

Linux Kernel

ClientSoftwareTools Server

Linux Kernel

PCI-ePCI

Packet Forwarding Table

NI

NetFPGA Router

VI VI VI VI

CG

E

GE

GE

GE

GE

GE

NetFPGA RouterHardware

5 Algo-Logic

EEEEEE

(eth1 .. 2)(nf2c0 .. 3)

Hardware & Software Components

f2 0 f2 1 f2 2 f2 3 i tlLinux Kernel

User-Space Software

nf2c0 nf2c1 nf2c2 nf2c3 ioctlLinux Kernel Driver

PCI Bus

CPU CPU nf2 reg grpCPU CPUCPU CPUFlip/FlopsCPU

RxQ TxQnf2_reg_grp

user data path

CPURxQ

CPUTxQ

CPURxQ

CPUTxQ

CPURxQ

CPUTxQ

NetFPGA Hardware:

Logic Gates,

& SRAMQueues

MACTxQ

MACRxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ

MACTxQ

MACRxQ

SRAM, & DRAMSRAM MAC

Queues,

6 Algo-LogicGigabit Ethernet

TxQ RxQ

PHY

Open Platform, Low-cost Hardware

• NetFPGA Cards & Pre-built Systems – Available from 3rd Party Vendors

• Or, build your system from parts– Details in the

on-line Guide

7 Algo-Logic

Stanford’s Rackmount Deployment

Statistics• Rack of 40

1U PC• 1U PCs • NetFPGAs

• Manged• Manged • Power,• Console• VLANs

• Provides 160 Gbps of full li tline-rate processing bandwidth

8 Algo-Logic

UCSD’s NetFPGA Cluster

9 Algo-Logic

Inter-module Communication

Table memory

data

y

data

ctrlwr

rdy

Packet memory

10 Algo-Logic

y

Modular Architecture

11 Algo-Logic

NetFPGA Base Reference Router

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

Input Arbiter

Output Port Lookup DRAMForward

TableLogic

PacketBufferLogic

SRAM

Output Queues

g g

MAC CPU MAC CPU MAC CPU MAC CPU

12 Algo-Logic

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Reference Router Control Software

Click the Details tab of the Quickstart windowthe Quickstart window

This is the reference router pipeline –a canonicala canonical, simple-to-understand, modular router pipelinep p

13 Algo-Logic

Control Software - Continued

Output Queues module (from Details tab)

Configuration details

Statistics

14 Algo-Logic

Enhanced Router Pipeline

Event Captureto capture output

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

queue events (writes, reads, drops)

Input Arbiter

O t t P t L kdrops) Output Port Lookup

Event Capture

Rate Limiterto create a

pOutput Queuesto create a bottleneck

MACTxQ

CPUTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

RateLimiter

15 Algo-Logic

MACTxQ

NetFPGA Hardware Block Diagram

NetFPGA platform

1M

18MSR

A1P

V2-Pro50 FPGA w/ infrastructureFo 1GE

M

AC

1GE

MA

C

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

Your hardware specifiedin Verilog source code

Mb

AM

1GE

PH

Y1G

EPH

Y

64MD

DR

SDR

A

ur Gigabit E E

C1G

E

MA

C1M

in Verilog source codeconnected to componentsof the Reference Routercircuits and cores.

-

E Y

1GE

PH

Y1P

MBR2AMEthernet Int

18Mb

SRA

M

Board- B1G

E

MA

C

1GE

PH

Y

terfaces

M

FIFOpacket

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

Control, PCIInterface

3 Gb

SATA

Board Interc

Linux OS - NetFPGA Kernel driverHostcomputer

User defined software networking applications

pbuffers Interface connect

16 Algo-Logic

User-defined software networking applications

Hardware Description LanguagesC• Concurrent– By default, Verilog statements

l t d tlevaluated concurrently

• Express fine grain parallelismExpress fine grain parallelism– Allows gate-level parallelism

• Provides Precise Description– Eliminates ambiguity about operation

• SynthesizableG t h d f d i ti

17 Algo-Logic

– Generates hardware from description

FPGA Look-Up TablesCombinatorial logic is stored

in Look-Up Tables (LUTs) – Also called

A B C D Z0 0 0 0 00 0 0 1 0Also called

Function Generators (FGs)– Capacity is limited only by

number of inputs, not complexity

0 0 0 1 00 0 1 0 00 0 1 1 1p , p y

– Delay through the LUT is constant 0 1 0 0 10 1 0 1 1

Combinatorial Logic

A

. . .1 1 0 0 01 1 0 1 0

B

CD

Z 1 1 1 0 01 1 1 1 1

18 Algo-Logic

Diagram From: Xilinx, Inc

Finite State Machines

Copyright 2001, John W. Lockwood, All Rights Reserved

Inputs (X)Outputs (Z) (X,S(t))

[Moore](S(t))

[Mealy]-or-

Combinational Logic

N tQ D

S(t) S(t+1)=(X,S(t)) State

Next

...Q D

State Storage

19 Algo-Logic

Field Programmable Gate ArraysDin Clk

CLB– Primitive element of FPGA

4 LUT

G4G3G2G1

G

4 LUT

F4F3F2F1

F

3 LUT

H

S

R

D Q

S

R

D Q

H1

YQ

Y

XQ

X

M

M

M

M

CLB

Routing Module– Global routing

Local interconnect

GRMLocal Routing

CLB PIP– Local interconnect

Macro Blocks 3rd Generation LUT-based FPGA

– Block Memories– Microprocessor ...

... ...

Macro

I/O Block ... ...

......

Block(uP,Mem)

20 Algo-Logic

Pad Routing CLB Matrix I/O

NetFPGA Reference Code & Modules

NF2 Directory Treebin (scripts for running simulations and setting up thebin

bitfiles

(scripts for running simulations and setting up the environment)

(contains the bitfiles for all projects that have been

lib

bitfiles (contains the bitfiles for all projects that have been synthesized)

(stable common modules and common partslib

j t

(stable common modules and common parts needed for simulation/synthesis/design)

projects (user projects, including reference designs)

21 Algo-Logic

Structure of Libray Source Code

lib

C ( ft d d f f d i )Cjava

(common software and code for reference designs)

(contains software for the graphical user interface)

MakefilesPerl5

(makefiles for simulation and synthesis)

(common libraries to interact with reference Perl5python

i t

(designs and aid in simulation)

(common libraries to aid in regression tests)

( i f f i )scriptsverilog

(scripts for common functions)

(modules and files that can be reused for design)

22 Algo-Logic

Structure of Project Source Code

projects

doc ( j t ifi d t ti )docinclude

(project specific documentation)(contains file to include verilog modules from lib, and creates project specific register defines files)

regresssrc

(regression tests used to test generated bitfiles)

(contains non-library verilog code used for th i d i l ti )src

swth

synthesis and simulation)(all software parts of the project)

(contains user xco files to generate cores andsynthverif

(contains user .xco files to generate cores and Makefile to implement the design)

(simulation tests)

23 Algo-Logic

Test-Driven Designs• Regression tests

– Have repeatable results – Define the supported features– Provide clear expectation on functionality

• Example: Internet Router– Drops packets with bad IP checksump p– Performs Longest Prefix Matching on destination address– Forwards IPv4 packets of length 64-1500 bytes

G t ICMP f k t ith TTL 1– Generates ICMP message for packets with TTL <= 1– Defines how packets with IP options or non IPv4

… and dozens more …

24 Algo-Logic

Every feature is defined by a regression test

Welcome to the Worldwide Community

NetFPGA @ SIGCOMM - Seattle, WA

Workshop in Beijing, China

NetFGPA @ SIGMETRICS - San Diego, CA

25 Algo-Logic

EuroSys - Glasgow, Scotland, U.K. Workshop in Bangalore, India

NetFPGA DeploymentsO 1 350 N tFPGA ith 1 300+ d d l d– Over 1,350 NetFPGA users with 1,300+ cards deployed

at 150+ universities in 17 Countries worldwide

26 Algo-Logic

Worldwide Hardware Deployments - Feb 2010

NetFPGA Hardware in North America

27 Algo-Logic

USA Deployments - Feb 2010

NetFPGA Hardware in Europe

28 Algo-Logic

European Deployments - Feb 2010

NetFPGA Hardware in Asia

China Korea Japan Taiwan and India

29 Algo-Logic

China, Korea, Japan, Taiwan, and India Deployments - Feb 2010

NetFPGA Deployments in Korea

KAIST

ETRI

CNU

30 Algo-Logic

Contributed projects

continued on next page

31 Algo-Logic.. And more on http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/ProjectTable

As of Feb 2010… continued on next page …

NetFPGA Designs (continued..)… continued from previous page …

32 Algo-Logic.. And more on http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/ProjectTable

Other Network FPGA Platforms

• NetFPGA 1G– 4 * 1 GigE– 4 Gbps

• FPX – 2 * OC-48– 4.8 Gbps

• Upcoming 10G– 4 * 10GE

33 Algo-Logic

– 40 Gbps

Motivation for Network Security • Viruses can be costly to businesses

– Annoyance to average users– Use networks to propagateUse networks to propagate

• Sample Attacks– Nimda, Code Red, Slammer, ,– MSBlast

• Infected over 350,000 hosts– SoBigFg

• Infected 1 million users in first 24 hours• Infected > 200 million in the first week• Caused an estimated $1 billion in damages to repair.

• End-systems difficult to maintain– Operating systems become outdated

Users introduce new machines on network

34 Algo-Logic

– Users introduce new machines on network

Example of SOC Firewall-FPX Platform

EM

SDSD

FPX Block Diagram FPX Photo

EM

SDSD

FPX Block Diagram FPX Photo

FlowB

uffer

Route

Filter

xtensibleM

odules

Layered Protocol Wrappers

RA

MSR

AM

RA

MSR

AM Memory

FlowB

uffer

Route

Filter

xtensibleM

odules

Layered Protocol Wrappers

RA

MSR

AM

RA

MSR

AM Memory

Cache

Program

Switch

Conf

NID (FPGA)

RAD (FPGA)

Cache

ProgramC

acheProgram

Switch

Conf

NID (FPGA)

RAD (FPGA)

• Reference

PRO

M

fig

Network Interface

PRO

M

fig

Network Interface

Reference– An Extensible, System-On-Programmable-Chip, Content-Aware Internet Firewall,

by John W. Lockwood, Christopher Neely, Christopher Zuver, James Moscola, Sarang Dharmapurikar, and David Lim; Field Programmable Logic and Applications (FPL) Lisbon Portugal Paper 14B Sep 1 3 2003

35 Algo-Logic

(FPL), Lisbon, Portugal, Paper 14B, Sep 1-3, 2003. • More Information:

– http://www.reprogrammablenetworks.com/

Regular Expression Matching

• String Matching– Allow for matching of fixed-length strings

• HEX(683063423739) finds SoBig.F( ) g

• Regular Expressions– Allows for a wide range of matches

• Case variations : (W|w)(A|a)(R|r)(H|h)(O|o)(L|l)• Wild-card characters: Albert ? Einstein • Strings of wildcards : A.* Einstein

Content may begin anywhere in packet– Content may begin anywhere in packet

• Morphable Patterns– Will require systems that can evolve– Will require systems that can evolve

36 Algo-Logic

GUI to Select Search Strings

37 Algo-Logic

Compiler to build Parallel Scan Engines

38 Algo-Logic

Content Matching Module

dataen out appl dataen_appl_in

RegularExpression

_ _ ppd_out_appl

sof_out_appleof_out_applsod out appl

_ _d_appl_insof_appl_ineof_appl_insod appl in

(RE)Matching

Circuit

sod_out_appltca_out_appl

sod_appl_intca_appl_in

Content clk

reset_lenable_l

MatchingVector

ready_l_

39 Algo-Logic

Sample ModelSim Waveform

TTL Src IP Dest IP

TTL <> 0 Src IP

40 Algo-Logic

Throughput Comparison• Sed was run on different Linux PCs

– Dual Intel Pentium III @ 1 GHz13 7 Mb h d t i d f di k• 13.7 Mbps when data is read from disk

• 32.72 Mbps when data is read from memory

– Alpha 21364 @ 667 MHz• 36 Mbps when data is read from disk• 50 4 Mbps when data is read from memory• 50.4 Mbps when data is read from memory

• Software results are 40x slower than FPsed

41 Algo-Logic

String Processing Benchmarks

42 Algo-Logic

Associative Match in FPGA TCAM

Src IP (hex) =80FC0505

Dest IP (hex) =8D8E0202

SrcPort = 1000

Dest Port =0050

Proto= 06

All values shown I h

Con-tent

03

C t t

80FC0505 8D8E0202 1000 0050 06 In hex= 03

• Content : Content Vector = “00000011” (binary) = x”03” (hex)

• Source IP Address : 128.252.5.5 (dotted.decimal)• Destination IP Address : 141 142 2 2 (dotted decimal)• Destination IP Address : 141.142.2.2 (dotted.decimal)• Source Port : 4096 (decimal) = 1000 (hex)• Destination Port : 80 (decimal) = 50 (hex)( ) ( )• Protocol : TCP (6)

43 Algo-Logic

TCAM Function in FPGA Hardware112 bits

CAM MASK [1]

CAM VALUE [1]

Flow ID [1]112 bits

16 bits

CAM MASK [2]

CAM VALUE [2]

Flow ID [2]

16 bitsCAM T blCAM MASK [3]

CAM VALUE [3]

Flow ID [3]Flow ID - - CAM Table - -

CAM MASK [N]

CAM VALUE [N]

Flow ID [N]

. . .. . .. . .

Resulting Flow

Identifier

Priority Encoder

Flow List

P l d Source Protocol

Bits in IP Header

44 Algo-LogicValue Comparators

Mask Matchers

Source Address Destination Address

Payload Match Bits

Source Port

Dest.Port

Protocol

Data Buffering & Per-flow Queuing

• Flow Buffering– Maintain linked list of packets– Track head and tail pointers for each list– Track free memory

• Queue Management– Enqueue packet– Dequeue packet

• Schedule Traffic Flows

45 Algo-Logic

Flow Buffering

A M[x]Flow State

Head Pointer

B M[y]Fi

Head Pointer

Head Pointer

Packet Reads= 0

C M[z]

iTail Pointer

Tail Pointer

Packet Writes= 3

M[u]OtherFlow

Head Pointer

Empty Slot

Reserved Empty Slot[implementation dependant]

[ ]FlowState

D M[v]

46 Algo-Logic

Queue Management

Flow BufferData

SRAMController

Queue ManagerTail NextTail HeadPacket Scheduler

NextHead

FlowID

P0

P1

DEnqueue

FSMP2

DequeueFSM

P3

47 Algo-Logic

Architecture of the SOC Firewall

Interfaces to Off-Chip Memories

Free List Manager

SRAM 1Controller

SDRAMController

SDRAM 2Controller

Xilinx XCV2000E FPGA

FlowBuffer

PayloadScanner

(FCCM’03)

TCAMFilter Extensible

Queue Manager

PacketScheduler

Payload Match Bits

(FCCM 03)

Flow ID

Module(s)

gPayload Match Bits Flow ID

Layered Internet Protocol Wrappers (FPL’01)

48 Algo-Logic

Resulting SOC-Firewall FPGA Layout

Per flowMemory

ControllerLayered Protocol Wrappers

Per-flow Queuing

sion

g

Man

ager

r Exp

ress

d Fi

lterin

Hea

der

ng t Sto

re M

Region for ExtensiblePlug-in Modules

Reg

ular

Payl

oad

TCA

M H

Filte

rin

Pack

et

49 Algo-Logic

Upcoming NetFPGA 10G (4 * 10 GE)QDRII+ SRAM

3x 36bit interfaces, 300MHz+(each i/f: 1x K7R643684MFC30)

XAUI 4 GTXs

PCIe 8 GTXs

SFI 10Gbps

SFP+ Cage

PCIe x8, Gen1 endpoint edge

connector

PHYAEL2005

Xilinx Virtex5

XCV5TX240T-2

XAUI 4 GTXs

XAUI

SFI 10Gbps

SFI

SFP+ Cage

SFP+

connector

PHY

PHYAEL2005

FG1759XAUI

4 GTXsSFI

10Gbps

SFI 10Gbps

SFP+ Cage

SFP+ Cage

10 GTXs

XAUI 4 GTXs

2 x Samtec x10 Connector10 GTXs

PHYAEL2005

PHYAEL2005

RLDRAM II2x 32bit interfaces 300MHz+

NetFPGA 10G

50 Algo-Logic

2x 32bit interfaces, 300MHz+

40 Gbps NetFPGA PCB

51 Algo-Logic

Algo-Logic Systems• Founded by Dr. John W. Lockwood, the

Algo-Logic ® team has extensive experience building routers, data center switches, and network processing circuits in ASICs and FPGA logic Algoprocessing circuits in ASICs and FPGA logic. Algo-Logic specializes in mapping network algorithms into hardware logic. The founders are experts in developing, documenting, and prototyping logic and systems of reprogrammable networksand systems of reprogrammable networks.

• Web– http://Algo-Logic com– http://Algo-Logic.com

• Email– Solutions@Algo-Logic.com

• PhonePhone– (650) 395-7026

• Fax– (650) 498-8296

• Office Address– 530 Lytton Ave

52 Algo-Logic

(650) 498 8296 ySecond FloorPalo Alto, CA 94301

Consulting Services• Design and Build Hardware-

Accelerated Network Systems

– Network Architecture • Data-center networks• Network security• Network security• Content-aware networks• On-chip interconnect

– Performance Analysis• Mathematical system modeling• Real and synthetic trace simulationy• Live measurements in hardware

prototypes

53 Algo-Logic

Design Services• Implementations of Network Algorithms

– Hardware Logic in synthesizable high-level HDLs, Verilog, and VHDLUltra low latency processing

Design, develop, and verify line-rate network

processing systems t lti Gb t– Ultra low latency processing

– System-level software withAPIs in C and C++

at multi-Gbps rates.

• Architectures for Next-Generation Networks– Data-center networks– Trading-floor networks – Network security– Network security– Content-aware networks

• Systems Architecture and Development– FPGA design – Verification and test bench development– Customization of IP cores– System-level Integration

54 Algo-Logic

System level Integration

Extended Training• Customized training

services to help your team build your own networkbuild your own network systems in hardware.

E i i t hi– Experience in teachingthe world-wide NetFPGA tutorials

C t i d t i i ifi ll– Customized training specifically tailored for your company’s skill level and requirements

– Hands-on training with live hardware systems

55 Algo-Logic

1st Asia NetFPGA Developer WorkshopJune 13-14, 2010 @ KAIST, Daejeon, Korea

http://fif.kr/AsiaNetFPGAws

• Program Committee Chairs– John W. Lockwood

• (Algo-Logic Systems)

• Advisory– Nick McKeown

• (Stanford University)– Seung-Joon Seok

• (Kyungnam University)

• Local Arrangement Co-Chairs

– Dae Young Kim • (Chungnam University)

• Many Thanksg– Sue Moon

• (KAIST)– Jaeyong Lee

• (Chungnam National Univ )

• Many Thanks– Xilinx– NetFPGA Group– Sponsors:• (Chungnam National Univ.) Sponsors:

56 Algo-Logic

Program Committee MembersY k L i

• Jun Bi – (Tsinghua University)

• Gordon Brebner

• Yukuen Lai – (Chung-Yuan

Christian Univ.)• Jaeyong Lee

• Seung-Yong Park– (Yonsei University)

• Hwangjun Song– (Xilinx)

• Michael Ciesla – (UNSW)

• Yu Huang Chu

y g– (Chungnam

National Univ.)• John W. Lockwood

(Algo Logic Systems)

– (POSTECH)• Charlie Wiseman

– (Washington University)• Yu-Huang Chu

– (Chunghwa Telecom)• Sun Moo Kang

– (NIA)

– (Algo-Logic Systems)• Kevin Xie

– (Xilinx)• Sue Moon

University)• Chong Ho Yoon

– (Korea Aerospace Univ.)( )

• Eric Keller – (Princeton University)

• Jongwon Kim (GIST)

– (KAIST)• Andrew, W. Moore

– (Cambridge University) Akihiro Nakao

• Martin Zadnik– (Brno University of

Tech.)• Hongyi Zeng– (GIST)

• Taekyoung Kwon – (Seoul National Univ.)

• Martin Labrecque

• Akihiro Nakao– (University of Tokyo)

• Seung Yeob Nam– (Yeungnam University)

Hongyi Zeng– (Stanford University)

• Young Cho – (USC-ISI)

57 Algo-Logic

Martin Labrecque– (University of Toronto)

( g y)• Seung-Joon Seok

• (Kyungnam University)

Welcome

78 Algo-Logic