嵌入式网络协议引擎 (ETA)加速网络应用程序处理

作者：林明峰

日期： 2003-12

概要• 今天的网络环境和网络操作系统• Linux 的 tcp/ip stack 的性能• 嵌入内核的 tcp 加速模块• ETA 的现实应用

今天的网络环境和操作系统

10G 的以太网络带来什么• 通用处理器必须每 67 奈秒处理一个数据包

• 超过通用处理器处理极限

• 协议占用太多时间

NP 是唯一的解决方案吗 ?

• Intel Ixp1200 带来好处

• 网络处理器对软件系统结果带来的问题

• 灵活性和高性能的矛盾

系统中断 内存拷贝缓冲区管理系统呼叫Tcp/ip 协议应用程序只能占用 3%

谁占用的 cpu 的时间 .

Linux 的 tcp/ip 协议

内核的 resource manager (ckrm)

• 控制 diskIO 的资源分配• 控制网络的 IO 分配• 目前还不成熟

• http://ckrm.sourceforge.net/

我们的解决方案• 区分出独立的协议处理层• 固定一个 cpu 来专门处理协议• 制定一个异步的通讯接口• 新的 socket 库来利用 ETA

ETA System Architecture

LAN Storage IPC

ETA Host Interface

IP StorageDriver

FileSystem Kernel

Applications

User Socket

Applications

SocketProxy

Network stackNetwork stack

Virtualized, Virtualized, asynchronous queuing asynchronous queuing and event handlingand event handling

Engine Architecture & Engine Architecture & platform integrationplatform integration

Network Fabric

Packet Processing Engine

Direct Transport Interface

ETAPacket Processing Engine

NIC

Application (Kernel or User)

Adaptation Layer

DTIEventQueue

DTIRx

Queue

DTITx

QueueAnonymous

BufferPool

DTIDoorbell

Shared Host Memory

AppBuffers

NIC…

DTI Operation Model• DTI operations:

•Connection requests (Connect, Listen, Bind, Accept, Close, …)•Data transfer requests (Send, Receive)•Misc. operations (Set/Get Options,…)

EVENT AEVENT BEVENT C

EventQTxQ

OP A

OP C

RxQ

OP B

OP D

DTIDoorbell

Process Operation

ServiceDoorbell

De-Queue OperationDescriptor

PostCompletion

Event

PostETA Interrupt

Event (if waiting)

Host Application

Adaptation layer

ETA PPESoftware

Gigabit NICs (5)

ETA Host Interface

KernelTest

Program

CPU 0Host

2.4 Ghz

CPU 1PPE

2.4 Ghz

Off-the-shelf Linux Servers

HostMemory Clients

ClientsClients

ClientsTestClients

KernelAbstraction Layer

ETA Test Environment

2P SMP Profile

App3%

Kernel97%

Idle

app3%

tcp/ip26%

bufmgt14%

driver20%

syscall7%

Interrupt10%

copy12%

other8%

idle Processing requirements in multiple components

TCP/IP is the largest single component, but is small compared to totalThe copy overhead is required to support legacy (synchronous) socket semanticsInterrupts and system calls are required in order to time-share the CPU resources

ETA Profile (1 host CPU + 1 PPE)

host app7%

host kernel3%

PPE51%

idle39%

host app8%

host kernel3%

PPE - tcp/ip12%

PPE - bufmgt7%

PPE - driver11%

PPE - Queuing12%

PPE - Other8%

Idle39%

Processing times are compressed

Idle time represents CPU resource that is usable for applicationsAsynchronous queuing interface avoids copy overheadInterrupts avoided by not time-sharing CPUSystem calls avoided by ETA queuing model

ETA2P SMP

Profile Comparisons

host app8%

host kernel3%

PPE - tcp/ip12%

PPE - bufmgt7%

PPE - driver11%

PPE - Queuing12%

PPE - Other8%

Idle39%

app3%

tcp/ip26%

bufmgt14%

driver20%

syscall7%

Interrupt10%

copy12%

other8%

idle

0.11

0.18

0.090.15

0.32

0.07

0.52

0.28

0.39

0.75

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

app tcp/ip bufmgt driver OTHER

CP

Us

Sp

ent

ETA

SMP

Normalized CPU Usage

ETA 的应用• 基于内核级别的 ids 程序• 独立于操作系统的包获取• 接管网卡