apic tutorial --- software: drivers, libraries and utilities john dehart washington university

June 16, 2002 APIC Tutorial 1WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

APIC

Tutorial---

Software: Drivers, Libraries and Utilities

John DeHartWashington University

[email protected]://www.arl.wustl.edu/~jdd


Overview

• Drivers– NetBSD

– Linux

– We will NOT talk about the MSR APIC Driver!!!

• User Mode Library• APICCell library• Utilities

– atm_ifconfig

– Traffic Generators


Driver Overview

• The Big Picture• IP• RATM


Driver Overview (Especially NetBSD)

APIC Kernel DriverENI Driver

ATM Device-Independent Driver

TCP/IPNativeATM

(NATM)

Socket Layer

TCP/IPApplication

User SpaceAPIC Driver

Appl.Signalling

Daemon

Controller

RemoteControl Lib.

atm_ifconfig

SignallingSocketSupport

SVCAppl.

APIC

Ker

nel

Use

r


IPv4 Forwarding in the NetBSD Kernel (APIC)

ipintrq:

ipintrip_forward ip_output

ip6_forward ip6_output

atmc_input

apic_isr apic_intr

ip6_input

atmc_output

input packets interrupts output packets

tcp/udp_input tcp/udp_output

socket layer

apic_devoutput

LINK

DATA

NETWORK

UPPER

LOWER

TRANSPORT


User IP in the NetBSD Kernel

ipintrq:

ipintrip_forward ip_output

atmc_input

apic_isr apic_intr

atmc_output

input packets interrupts output packets

tcp/udp_input tcp/udp_output

socket layer

apic_devoutput

LINK

DATA

NETWORK

UPPER

LOWER

User Program

SoftwareInterrupt

Waits in SocketBuffer for User

Program to pick up

TRANSPORT


Two Sets of Receive Descriptors/Buffers

...

AAL5 Receive Pool ChainC

lust

erm

bu

f

Clu

ster

mb

uf

Clu

ster

mb

uf

DDD

Host Memory

...

AAL0 Receive Pool Chain

Clu

ster

mb

uf

Clu

ster

mb

uf

Clu

ster

mb

uf

DDD

Host Memory

- % 48 => - % 56 =>C

lust

erm

bu

f2048

Currently the same size,but not necessarily so

Clu

ster

mb

uf

2048

AA

L5

bu

ffer

2016

AA

L0

bu

ffer

2016


Interrupt on Done

APIC

Interrupt

Driver ReadsWhich Channel

Caused Intr.

Chan N

AAL5 Frame

Host Memory

Notify RegisterChannel N

IP QEOF

Done Done Done


Interrupt on Descriptor Read

APIC

Interrupt

Driver ReadsWhich Channel

Caused Intr.

Chan N

Host Memory

Notify RegisterChannel N

RATM Q

...

Receive Pool ChainC

lust

erm

bu

f

Clu

ster

mb

uf

DD

Host MemoryCurrent

Descriptor

APIC Reads NextDescriptor & Makes it

Current Descriptorfor Chan N

Done


Receive Pools: Two Per-Frame, One Per-Cell

• Two Per-Frame Pools:– AAL5

• Interrupt on DONE• size <= requested system buffer (mbuf) size• size % 48 = 0• Used for IP and RATM AAL5 applications

– AAL0• Interrupt on Descriptor Read• size <= requested system buffer (mbuf) size• size % 56 = 0• Used for continuous AAL0 data applications

– For current cluster mbuf• AAL0_size = AAL5_size = 2016

– Fairly large number of buffers (100’s to 1000’s)


Receive Pools (con’t)

• AAL0 per-cell Pool– buffer size = 56 bytes

– Interrupt on Descriptor Read• Each time APIC reads a new Descriptor, the previous one is

available for the Driver to process

– Used for per-cell AAL0 applications like GBNSC

– Small number of buffers (10’s)

– CAVEAT: Deadly if used for high bandwidth channel• Interrupt on EVERY Cell

– We are using a cluster mbuf (2K Bytes) for each cell!!

• Can we use regular mbuf (~200 Bytes)?

• Physical address alignment implications?


atm_ifconfig

• NetBSD ATM VC configuration utility> atm_ifconfigUsage: atm_ifconfig <interface> closeall Usage: atm_ifconfig <interface> getall Usage: atm_ifconfig <interface> <vpivci> [open|close|modify [options...]]Options are: -aal0 | -aal0_cell | -aal5 -llc | -nollc <protocol>: use LLC/SNAP? protocol is inet | ratm | value of ethernet type -loopback -inport [FIBER,RIBBON,SWITCH,LINK] -outports [FIBER,RIBBON,SWITCH,LINK,BOTH] -lowdelay | -paced <rate> | -besteffort -maxqueue <max_descriptors_in_tx_queue> rate is in Kbits/sec


NetBSD Device, VC and Route Configuration ifconfig apic0 inet 192.168.208.2 netmask 0xffffff00 up

• Configure VCI 32 for newGBNSC to use (NetBSD): atm_ifconfig apic0 0x20 open -aal0_cell -nollc ratm -inport FIBER -outports FIBER

• Configure VCI 100 for IP usage (NetBSD): atm_ifconfig apic0 0x64 open –aal5 -nollc inet -inport FIBER -outports FIBER route add –iface 192.168.208.1 -link apic0:0.0.0.64

Error on handouts


Linux Device, VC and Route Configuration

• /etc/sysconfig/network-scripts/ifcfg-atm0• /etc/init.d/atm start• atmsigd• ilmid• atmarp• route add


Linux Device, VC and Route Configuration (con’t)#

# /etc/sysconfig/network-scripts/ifcfg-atm*

#

DEVICE=atm0 # device name wuapic0

DEVICE_ADDR=demand4-apic0 # host's hardware address (made up)

IPADDR=192.168.10.3 # hosts's IP address on the ATM LIS

NETMASK=255.255.255.0 # netmask (e.g. arl uses 8 host bits)

NETWORK=192.168.10.0 # network (e.g. arl's experimental ATM LIS)

BROADCAST=192.168.10.255 # broadcast address (not really useful)

# ARPSRV=lrcsuns-a # ATM address of ATMARP server; omit if local

# ARPSRVQOS="ubr:max_pcr=100kbps" # QOS of the ATMARP server VC

# ARPDEFQOS="ubr:max_pcr=700kbps" # default QOS for IP over ATM VCs

ONBOOT=no # needs to be loaded by ATM-specific procedure

MTU=32768 # larger packets ok on tx


Linux Device, VC and Route Configuration (con’t)> /etc/init.d/atm startStarting ATM demons: [OK]

> ps -auxwww | grep atmsigdroot 973 0.0 0.1 1708 628 pts/1 S 09:56 0:00 atmsigd -b -l

syslog -D /var/tmp -t 20root 1019 0.0 0.1 1708 600 pts/1 S 09:56 0:00 grep atmsigd

> ps -auxwww | grep ilmidroot 984 0.0 0.0 1428 384 pts/1 S 09:56 0:00 ilmid -b -l syslogroot 1021 0.0 0.1 1708 600 pts/1 S 09:56 0:00 grep ilmid

# YOU MAY WANT TO KILL THESE… kill 973 984• atmsigd and ilmid are started in /etc/init.d/atm

– If you are SURE you don’t want them on your machines back home, that is the place to edit them out. DO NOT do that here!!!


Linux Device, VC and Route Configuration (con’t)

> ifconfig atm0

atm0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00

inet addr:192.168.10.3 Mask:255.255.255.0

UP RUNNING MTU:32768 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:100


Linux Device, VC and Route Configuration (con’t)> atmarp -s 192.168.10.1 0.0.50 null

# for a network route:> route add -net 192.168.10.0 netmask 255.255.255.0 gw 192.168.10.1# for a host route:> route add –host 192.168.10.2 gw 192.168.10.1

> atmarp -a----- Itf 0 (192.168.10.3, netmask 255.255.255.0) -----Default QOS: ubr,aal5:max_sdu=9188IP 192.168.10.1, state VALID, addr <none>, flags 0x1004<PERM,NULL> 0.0.50 Send buffer: 65535----- Unknown incoming connections ---------- Incoming unidirectional connections ---------- End of dump -----


Linux Device, VC and Route Configuration (con’t)> netstat -nr

Kernel IP routing table

Destination Gateway Genmask Flags MSS Window irtt Iface

128.252.153.0 0.0.0.0 255.255.255.0 U 40 0 0 eth0

192.168.10.0 192.168.10.1 255.255.255.0 UG 40 0 0 atm0

192.168.10.0 0.0.0.0 255.255.255.0 U 40 0 0 atm0

192.168.10.0 0.0.0.0 255.255.255.0 U 40 0 0 atm0

127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo

0.0.0.0 128.252.153.249 0.0.0.0 UG 40 0 0 eth0


User Mode Library

• User Mode library is Driver functionality exposed to User Space programs.– You get direct manipulation of Descriptors and Buffers

– Access to APIC Registers

– It has many of the complexities of a Driver.

– It has many of the DANGERS of a Driver.

• Source: wu_arl/utilities/APIC/ApicLibrary• Binary: wu_arl/utilities/lib/$(OSTYPE)/libAPIC.a• Installed include files: wu_arl/utilities/include/


User Mode Library

• Following slides contain code from one of our utility programs: wu_arl/utilities/APIC/FileTransfer/FileRead.c

• Best way to learn how to use the User Mode library is to start from existing code.

• If you want to use the User Mode library:YOU MUST INVEST SOME TIME!!!

• Start by using the FileTransfer utility. – Get it to work for you.– Add your own debug statements to it– Figure out what it is doing.– Then use it as an example to do what you want.


User Mode library example

/* From wu_arl/utilities/APIC/FileTransfer/FileRead.c *//* Initialize */

i = InitApicLibrary(false); /* try for an apic */ if (i) { fprintf(stdout, "Failed to grab an apic - is the user mode code enabled?\n"); exit(-1); }

/* map the Data (main data pool) */ ApicHandle = ApicLibraryMap(0, /* use default apic id */ APIC_LIBRARY_READ5, /* open in read mode */ i, /* initial descr */ FramesReady, /* buffer count */ 4096, /* size of frames */ rvpi, /* VPI connection */ rvci, /* VCI connection */ ENDIANNESS_LITTLE, /* endian (little) */ 0x0000); /* do not pack to page boundary */ if (!ApicHandle) { perror("Failed to init DataPool"); exit(-1); }



/* From wu_arl/utilities/APIC/FileTransfer/FileRead.c */

/* Start reading */ AckHandle = ApicHandle; /* save output vci data */

if (ApicHandle->udescr <= 0)

{

fprintf(stdout, "No AckPool descriptors allocated - see /var/log/messages\n");

exit(-1);

}

AckFirstDescr = AckHandle->udescr; /* obtain first used descriptor */

AckCount = AckHandle->udescc;

AckLastDescr = AckFirstDescr + AckCount - 1;

bzero(AckHandle->ubuf, 2 * page_size);

i = ApicWriteDescriptor0(AckFirstDescr + 1, NotReady, AckHandle);

NextDescr = DataFirstDescr; /* init circular chain */

i = ApicStartAAL5Read(DataHandle); /* always read first */

fprintf(stdout, "Looking for FileSend AAL5 data\n");

SendAckNak(-1, ACKNAK_IDLE1); /* send an I'm Alive */




/* Processing loop. Process36K() does the work */ while (!kern_start_addr)

{

int what = Process36K();

if (what < 0)

{

if (Debug)

fprintf(stdout, "failure code %d\n", what);

if (LastSequence == -1) {

// we still haven't received our first frame. Continuing sending pings

SendAckNak(-1, ACKNAK_IDLE1); /* send an I'm Alive */

}

else

SendAckNak(LastSequence, what); /* ping the other side */

continue;

}

SendAckNak(LastSequence, ACKNAK_OK); /* say ok to the other side */

}



/* From wu_arl/utilities/APIC/FileTransfer/FileRead.c *//* In Process36K(): Check for a DONE_VALID descriptor */

/* patrol the descriptors for this i/o */

while (!CheckReady(NextDescr)) /* (1:36K, 1:4k) X 8 segments */ { w0 = ApicReadDescriptor0(NextDescr, DataHandle); w1 = ApicReadDescriptor1(NextDescr, DataHandle); if (apic_fval(w0, DESC_SYNC) != DESC_SYNC_DONE_VALID_LINK) { if (Debug) { fprintf(stdout,

"Not Valid Next Descr:0x%04x, W0:0x%08x, W1:0x%08x\n",

NextDescr, w0, w1); } return retcode; }




/* In Process36K(): Examine data */

/* we have a valid well aligned frame */

Trailer = (unsigned int *) ((unsigned char *) DataHandle->ubuf + offset * 4096); /* get real address */

LT = (struct load_trailer *) Trailer; /* make a struct of it */

Bytes = LT->DataSize0;

Swap = LT->ByteSwap0; /* get byte swap code */

Dest = (unsigned int *) LT->PAddress0;

LastSequence = LT->Sequence0;



/* From wu_arl/utilities/APIC/FileTransfer/FileRead.c *//* In Process36K(): Write Data to a file */

if (fd) { int *s1 = (int *) ((unsigned char *) DataHandle->ubuf + ((offset - APIC_AAL5_PAYLOAD_PAGES) *

4096)); int *s2 = s1; int q = 0; for (q = 0; q < 512 * 8; q++) /* swap words for output */ {

int x = *s2; *s2 = *(s2 + 1); *(s2 + 1) = x; s2 += 2; } out = write(fd, s1, LT->DataSize0); }


AAL5Generator• Uses APIC User Mode Library

– Configures one APIC descriptor/buffer pair• Sets descriptor to point to itself and be Read-Only

• INFINITE LOOP!! (but it’s a good infinite loop)

• APIC will transmit the buffer until descriptor is turned off

– After a given length of time (approximate!!)• Turns descriptor off

– Uses APIC global pacer and per VC pacing• controls bit/sec rate and packet/sec rate

• Intended for high performance test and analysis

• Multiple instances can run at same time on same machine

• Companion sendCommand utility– to change characteristics on the fly

• Need to add control for APIC Batch size– APIC batches cells transmitted. Controlled by an APIC register

– easy to add control, just has not been done yet.


sendCommand

• (Different then the MSR sendcmd you will see tomorrow)

• Provides remote control of AAL5Generator– changes the charateristics of a running AAL5Generator

• What can be changed:– Packet Rate

– Kbit Rate

– Destination Address

– Run State:• Quit

• Stop

• Start


sendCommand

• Sample Commands:– Sends the command to AAL5Generator on tabby which is

listening on port 4444 for incoming commands:

– Change the packet per second rate to 10000: sendCommand –p 4444 tabby “Packet Rate 10000”

– Change the Kbit per second rate to 10000: sendCommand –p 4444 tabby “Kbit Rate 10000”

– Change the Destination IP Address to 192.168.212.2: sendCommand –p 4444 tabby “DestAddr 192.168.212.2”

– Stop sending packets: sendCommand –p 4444 tabby “Stop”

– Restart sending packets: sendCommand –p 4444 tabby “Start”

– Quit: causes AAL5Generator to Stop sending and exit: sendCommand –p 4444 tabby “Quit”


Source and Executables

• source location: – wu_arl/utilities/APIC/AAL5Generator

• binary location:– wu_arl/utilities/bin/$OSTYPE/AAL5Generator

– wu_arl/utilities/bin/$OSTYPE/sendCommand


AAL5Generator Usage

Most Used Command Line Options:[-svci #] (VCI for local sending of AAL5 Frames)

[-kbits #] (xmit rate using paced link rate)

[-frate #] (Frames/sec)

[-seconds #] (running time, 0 means run forever)

[-packet #] (20 <= X <=1912) pkt size (bytes) includes IP hdr

[-port #] (UDP port on which to receive ctrl commands)

[-src #] (IP Source address to put in packet IP Header)

[-dst #] (IP Destination address to put in packet IP Hdr)


AAL5Generator Usage (cont’d)

Less Used Command Line Options:[-pacer #] (0:88Mb/S <= X <= 255:3.5Mb/S pacer clock)

[-svpi #] (VPI for local sending of AAL5 Frames (0))

[-rate #] (16 bit divisor whole)

[-frac #] (8 bit divisor fractional)

[-fsize #] (0<=X<=7) (40,136,232,520,760,1192,1480,1912)

[-descr #] (0 < X < 64K initial descr)

[-debug ] (print debug info)

[-swap ] (htonl() byte swap output)

[-apic #] (Linux: Apic Id (0 <= X <= 3))


APICCell Library API

• Provides a cell manager for sending and receiving APIC control cells

• Uses the GBNCELL_CellManager– for now it assumes we are talking to a WUGS– thus it is used to talk to SPCs

• Provides classes for APIC Registers• Provides member functions for reading and

writing:– External Memory– Global Registers– Transmit Registers– Receive Registers


APICCell Library Files

• Source location:– wu_arl/wugs/src/common_code/APICCELL

• Binary:– wu_arl/wugs/lib/$(OSTYPE)/libAPICCELL.a

• Sample test code:– wu_arl/wugs/src/common_code/APICCELL/test_code


APICCell Library Example// Class defn from wu_arl/wugs/src/common_code/APICCELL/test_code/RWPacingParam.hclass RWPacingParameter : public APIC_OperationBase{private: APIC_CellManager* cellManager; Boolean done; Boolean isRead;public: RWPacingParameter(APIC_CellManager* cm) : APIC_OperationBase() { cellManager = cm; done = FALSE; isRead = TRUE; } void readParameter(); void writeParameter(int p); virtual void failed(APIC_ControlCell& c); virtual void receiveCell(APIC_ControlCell& c); Boolean isDone() { return done;}};


APICCell Library Example

// Snippet of code from

// wu_arl/…./APICCELL/test_code/RWPacingParam.Cvoid

RWPacingParameter::writeParameter(int p)

{

isRead = FALSE;

done = FALSE;

APIC_ControlCell cell;

APIC_RegisterData* tmp_data = cell.writeGlobalRegister (

APIC_RegID_PacingParameter, // write this global apic register

TRUE, //cell should return

NULL, //data to send NULL causes one to be a data

// structure to be created and returned to us

getCMData()); //and this CMDATA value

tmp_data->setPacingParameter(p);

cellManager->send(cell);

}


APICCell Library Example

// Snippet of code from

// wu_arl/…./APICCELL/test_code/RWPacingParam.Cvoid

RWPacingParameter::receiveCell(APIC_ControlCell &cell)

{

APIC_RegisterData* data = cell.getData();

cout << endl << "\t" << data->getName() ;

if (isRead)

{

cout << "\t = " << data->getPacingParameter() << endl;

}

else cout << ": \t returned " << endl;

done = TRUE;

}


Notes

apic tutorial --- software: drivers, libraries and utilities john dehart washington university

Documents