suricata and xdp, performance with a s like security · kernel ask userspace for decision on...

Suricata and XDP, Performance with a S like Security

É. Leblond

Nov. 29, 2018

É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 1 / 43

1 IntroductionSuricata 101Suricata on live traffic

2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more

3 Suricata BypassIntroducing bypassBypass strategy

4 Extended Berkeley Packet Filter5 AF_PACKET bypass

eBPF bypassXDP support

6 Conclusion

Who am I?

Éric LeblondStamus Networks co-founder

Editor of a threat hunting solutionincluding Suricata based appliances

Netfilter core team memberReally low personal activity nowadays

eleblond@oisf.netLong time member of OISFSuricata developer

In charge of packet acquisition1118 commits since 2010 (I like small patches)

About the journey

Adding bypass feature to Suricata2 years of development to see less and get more doneUsing kick ass technologies before their documentation has been written.

Figure: Summary of talk objectives

6 Conclusion

What it is not ?

https://twitter.com/randomuserid/status/1012474246503845888É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 4 / 43

A signature based IDS

From individual datagram to detectionGet packet per packetReconstruct to application layerRun detection engine

Key pointsGPLv2Owned by OISF foundation10 years oldScalability via multithreadingWritten in C and Rust

Example signature

Suricata NSM features

Supported protocolsProtocol analysis: http, ftp, smtp, tls, ssh smb, dcerpc, dns, nfs, ntp, ftp-data, tftp,ikev2, krb5, dhcpProtocol recognition: imap, msn

Log example

What it is ? or how to please developers

https://twitter.com/randomuserid/status/1012705279098490880

File related features

File analysisMagic computation and in file data matchChecksum computation and file extraction to diskSupported protocols: http, smtp, smb, ftp, nfs

Fileinfo example

6 Conclusion

Suricata live modes

Intrusion Detection SystemAF_PACKET capture method under LinuxGet raw packet from cardDo complete analysis

Intrusion Prevention System

Netfilter with NFQUEUE on LinuxKernel ask userspace for decision onpackets

6 Conclusion

Suricata reconstruction and normalization

6 Conclusion

Impact of loosing packets

MethodologyUse a sample trafficModify the pcap file to have specified random packet lossDo it 3 times par packet lossGet graph out of that

Test dataUsing a test pcap of 445Mo.Real traffic but lot of malicious behaviorsTraffic is a bit old

Alert loss by packet loss

Some numbers10% missed alerts with 3% packets loss50% missed alerts with 25% packets loss

The case of file extraction

Some numbers10% failed file extraction with 0.4% packets loss50% failed file extraction with 5.5% packets loss

6 Conclusion

The elephant flow problem (1/2)

The elephant flow problem (2/2)

Ring buffer overrunLimited sized ring bufferOverrun cause packets lossthat cause streaming malfunction

Ring size increaseWork aroundUse memoryFail for non burst

Dequeue at NQueue at speed N+M

6 Conclusion

Stream depth method

Attacks characteristicIn most cases attack is done at start of TCP sessionGeneration of requests prior to attack is not commonMultiple requests are often not even possible on same TCP session

Stream reassembly depthReassembly is done till stream.reassembly.depth bytes.Stream is not analyzed once limit is reachedIndividual packet continue to be inspected

6 Conclusion

Introducing bypass

Stop packet handling as soon as possibleTag flow as bypassedMaintain table of bypassed flowsDiscard packet if part of a bypassed flow

Bypass methodLocal bypass: Suricata discard packet after decodingCapture bypass: capture method maintain flow table and discard packets ofbypassed flows

Bypassing big flow: local bypass

Bypassing big flow: capture bypass

Implementation

Suricata updateAdd callback functionCapture method register itself and provide a callbackSuricata calls callback when it wants to offload

NFQ bypass in Suricata 3.2Update capture register functionWritten callback function

Set a mark with respect to a mask on packetMark is set on packet when issuing the verdict

Suricata NFQ and bypass

6 Conclusion

Stream depth bypass

Stop all treatment after bypassGo beyond what is currently doneDisable individual packet treatment once stream depth is reached

Activating stream depth bypassSet stream.bypass to yes in YAML

TLS bypassencrypt-handling: bypass

Selective bypass

Ignore some trafficIgnore intensive traffic like NetflixCan be done independently of stream depthCan be done using generic or custom signatures

The bypass keywordA new bypass signature keywordTrigger bypass when signature matchExample of signature

pass h t t p any any −> any any ( content : " s u r i c a t a . i o " ; \ \h t tp_hos t ; bypass ; s id :6666; rev : 1 ; )

And now AF_PACKET bypass

What’s neededSuricata to tell kernel to ignore flowsKernel system able to

Maintain a list of flow entriesDiscard packets belonging to flows in the listUpdate from userspace

eBPF filter using mapseBPF introduce mapsDifferent data structures

Hash, array, . . .Update and fetch from userspace

Looks good!

6 Conclusion

Extended Berkeley Packet Filter

Berkeley Packet FilterVirtual machine inside kernelArithmetic operations and tests on the packet dataFilters are injected by userspace in kernel via syscall

Extended BPFExtended virtual machine: more operators, data and function accessVarious attachment points

SocketSyscallTraffic control

Kernel and userspace shared structuresHash tablesArrays

LLVM backend

From C file to eBPF codeWrite C codeUse eBPF LLVM backend (since LLVM 3.7)Use libbpf

Get ELF fileExtract and load section in kernel

BCC: BPF Compiler collectionInject eBPF into kernel from high level scripting languageTrace syscalls and kernel functionshttps://github.com/iovisor/bcc

BCC tracing tools

http://www.brendangregg.com/ebpf.htmlÉ. Leblond (OISF) Suricata and XDP Nov. 29, 2018 27 / 43

eBPF applied to security

AdvantagesReally extensibleKernel version independant when not intercepting functionsExtract info all system stacks

Host security monitoring at NetflixLinux Monitoring at Scale with eBPF (Brendan Gregg & Alex Maestretti)https://youtu.be/44nV6Mj11uw

Example: BCC socket bind 1/2

Detect network serversGet all bind call to detect servicesOutput result to console

A BCC scriptPython codeeBPF code as C in a string

Example: BCC socket bind 2/2

DemoStart sobindStart a nc command to listen to port 2233

Outputsudo python ./sobindPID COMM PROTO PORT ADDR9565 nc TCPv4 2233 0.0.0.09572 nc TCPv4 2233 127.0.0.2

Key featuresDirect hook in the system callNo /proc scanning but polling of resultsGet it there:https://gist.github.com/regit/1e591311fa3ba5cd0b8d73940348599a

6 Conclusion

Suricata eBPF bypass architecture

Test methodology

Test setupIntel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHzIntel Corporation 82599ES 10-Gigabit SFI/SFP+Live traffic:

Around 1Gbps to 2GbpsReal users so not reproducible

TestsOne hour long runDifferent stream depth valuesCollected Suricata statistics counters (JSON export)Graphs done via Timelion(https://www.elastic.co/blog/timelion-timeline)

Results: stream bypass at 512kb

A few words on graphics

Tests at 512kbWe have on big flow that kill thebandwidthCapture get almost nullEven number of closed bypassedflows is low

Results

6 Conclusion

A Linux kernel feature

Run a eBPF code the earliest possiblein the driverin the cardbefore the regular kernel path

Act on dataDrop packet (eXtreme Drop Performance)Transmit to kernelRewrite and transmit packet to kernelRedirect to another interfaceCPU load balance

Implementation

Similar to eBPF filterSame logic for bypassOnly verdict logic is different

But annoying differenceeBPF code does the parsingNeed to bind to an interface

Suricata XDP architecture

AF_PACKET IPS mode

IPS and bypassCan’t drop packet at bypass stageNeed to forward from one iface to another

XDP and IPS mode: bypass

Use TX_REDIRECTDirect copy from interface to interface

Results

Direct NIC to NIC transferSkip all kernel taskWire speed copyIf eBPF code is fast enough

Obtained performanceTODO: Ask OISF marketing for some fake numbers to show

6 Conclusion

Conclusion

Suricata, eBPF and XDPAvailable in Suricata 4.1, need Linux 4.16Network card bypass for Netronome comingAF_XDP capture is now in Linux vanilla

More informationStamus Networks: https://www.stamus-networks.com/Septun II: https://github.com/pevma/SEPTun-Mark-II/Suricata doc: http://suricata.readthedocs.io/en/latest/capture-hardware/ebpf-xdp.html

Questions ?

Thanks toJesper Dangaard BrouerAlexei StarovoitovDaniel Borkmann

Contact meeleblond@oisf.netTwitter: @regiteric

Want more fun ?Come to Suricata trainings: https://suricata-ids.org/training/

Suricon: https://suricon.net/

suricata and xdp, performance with a s like security · kernel ask userspace for decision on...

Documents

suricata - workshop.netfilter.org · suricata Éric leblond...

extreme performance - suricon...xdp & suricata code work...

suricata ids/ips - vereniging nluug · suricata creator and...

bro befriends suricata - the bro network security … ·...

bro befriends suricata - zeek.org · bro befriends suricata...

eric leblond ids-suricata

suricata ips mode

suricata, the terminator of ids/ips world · suricata, the...

why ebpf and xdp in suricata matters · 2019. 1. 9. ·...

amperis manual xdp ii

suricata & aws - 2020 suricon in boston presented by oisf...

oisf: regular expressions (regex) overview

isa xdp (experiential design program) xdp 2019... · isa...

meerkat (suricata suricatta)

suricata - suricata.yaml reference

xdp - express data path · 2017-11-03 · 5/27 xdp...

suricata suricatta

xdp - express data path xdp now with...

who am i? past: commercial waf developer since 2007...

a short introduction to suricata idps -...