suricata and xdp, performance with a s like security · kernel ask userspace for decision on...

Suricata and XDP, Performance with a S like Security

É. Leblond

OISF

Nov. 29, 2018

É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 1 / 43

1 IntroductionSuricata 101Suricata on live traffic

2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more

3 Suricata BypassIntroducing bypassBypass strategy

4 Extended Berkeley Packet Filter5 AF_PACKET bypass

eBPF bypassXDP support

6 Conclusion


Who am I?

Éric LeblondStamus Networks co-founder

Editor of a threat hunting solutionincluding Suricata based appliances

Netfilter core team memberReally low personal activity nowadays

[email protected] time member of OISFSuricata developer

In charge of packet acquisition1118 commits since 2010 (I like small patches)


About the journey

Adding bypass feature to Suricata2 years of development to see less and get more doneUsing kick ass technologies before their documentation has been written.

Figure: Summary of talk objectives







6 Conclusion


What it is not ?

https://twitter.com/randomuserid/status/1012474246503845888É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 4 / 43

https://twitter.com/randomuserid/status/1012474246503845888

A signature based IDS

From individual datagram to detectionGet packet per packetReconstruct to application layerRun detection engine

Key pointsGPLv2Owned by OISF foundation10 years oldScalability via multithreadingWritten in C and Rust

Example signature


Suricata NSM features

Supported protocolsProtocol analysis: http, ftp, smtp, tls, ssh smb, dcerpc, dns, nfs, ntp, ftp-data, tftp,ikev2, krb5, dhcpProtocol recognition: imap, msn

Log example


What it is ? or how to please developers




File related features

File analysisMagic computation and in file data matchChecksum computation and file extraction to diskSupported protocols: http, smtp, smb, ftp, nfs

Fileinfo example







6 Conclusion


Suricata live modes

Intrusion Detection SystemAF_PACKET capture method under LinuxGet raw packet from cardDo complete analysis

Intrusion Prevention System

Netfilter with NFQUEUE on LinuxKernel ask userspace for decision onpackets







6 Conclusion


Suricata reconstruction and normalization







6 Conclusion


Impact of loosing packets

MethodologyUse a sample trafficModify the pcap file to have specified random packet lossDo it 3 times par packet lossGet graph out of that

Test dataUsing a test pcap of 445Mo.Real traffic but lot of malicious behaviorsTraffic is a bit old


Alert loss by packet loss

Some numbers10% missed alerts with 3% packets loss50% missed alerts with 25% packets loss


The case of file extraction

Some numbers10% failed file extraction with 0.4% packets loss50% failed file extraction with 5.5% packets loss







6 Conclusion


The elephant flow problem (1/2)


The elephant flow problem (2/2)

Ring buffer overrunLimited sized ring bufferOverrun cause packets lossthat cause streaming malfunction

Ring size increaseWork aroundUse memoryFail for non burst

Dequeue at NQueue at speed N+M







6 Conclusion


Stream depth method

Attacks characteristicIn most cases attack is done at start of TCP sessionGeneration of requests prior to attack is not commonMultiple requests are often not even possible on same TCP session

Stream reassembly depthReassembly is done till stream.reassembly.depth bytes.Stream is not analyzed once limit is reachedIndividual packet continue to be inspected







6 Conclusion


Introducing bypass

Stop packet handling as soon as possibleTag flow as bypassedMaintain table of bypassed flowsDiscard packet if part of a bypassed flow

Bypass methodLocal bypass: Suricata discard packet after decodingCapture bypass: capture method maintain flow table and discard packets ofbypassed flows


Bypassing big flow: local bypass


Bypassing big flow: capture bypass


Implementation

Suricata updateAdd callback functionCapture method register itself and provide a callbackSuricata calls callback when it wants to offload

NFQ bypass in Suricata 3.2Update capture register functionWritten callback function

Set a mark with respect to a mask on packetMark is set on packet when issuing the verdict


Suricata NFQ and bypass







6 Conclusion


Stream depth bypass

Stop all treatment after bypassGo beyond what is currently doneDisable individual packet treatment once stream depth is reached

Activating stream depth bypassSet stream.bypass to yes in YAML

TLS bypassencrypt-handling: bypass


Selective bypass

Ignore some trafficIgnore intensive traffic like NetflixCan be done independently of stream depthCan be done using generic or custom signatures

The bypass keywordA new bypass signature keywordTrigger bypass when signature matchExample of signature

pass h t t p any any −> any any ( content : " s u r i c a t a . i o " ; \ \h t tp_hos t ; bypass ; s id :6666; rev : 1 ; )


And now AF_PACKET bypass

What’s neededSuricata to tell kernel to ignore flowsKernel system able to

Maintain a list of flow entriesDiscard packets belonging to flows in the listUpdate from userspace

eBPF filter using mapseBPF introduce mapsDifferent data structures

Hash, array, . . .Update and fetch from userspace

Looks good!







6 Conclusion


Extended Berkeley Packet Filter

Berkeley Packet FilterVirtual machine inside kernelArithmetic operations and tests on the packet dataFilters are injected by userspace in kernel via syscall

Extended BPFExtended virtual machine: more operators, data and function accessVarious attachment points

SocketSyscallTraffic control

Kernel and userspace shared structuresHash tablesArrays


LLVM backend

From C file to eBPF codeWrite C codeUse eBPF LLVM backend (since LLVM 3.7)Use libbpf

Get ELF fileExtract and load section in kernel

BCC: BPF Compiler collectionInject eBPF into kernel from high level scripting languageTrace syscalls and kernel functionshttps://github.com/iovisor/bcc


https://github.com/iovisor/bcc

BCC tracing tools

http://www.brendangregg.com/ebpf.htmlÉ. Leblond (OISF) Suricata and XDP Nov. 29, 2018 27 / 43

http://www.brendangregg.com/ebpf.html

eBPF applied to security

AdvantagesReally extensibleKernel version independant when not intercepting functionsExtract info all system stacks

Host security monitoring at NetflixLinux Monitoring at Scale with eBPF (Brendan Gregg & Alex Maestretti)https://youtu.be/44nV6Mj11uw


https://youtu.be/44nV6Mj11uw

Example: BCC socket bind 1/2

Detect network serversGet all bind call to detect servicesOutput result to console

A BCC scriptPython codeeBPF code as C in a string


Example: BCC socket bind 2/2

DemoStart sobindStart a nc command to listen to port 2233

Outputsudo python ./sobindPID COMM PROTO PORT ADDR9565 nc TCPv4 2233 0.0.0.09572 nc TCPv4 2233 127.0.0.2

Key featuresDirect hook in the system callNo /proc scanning but polling of resultsGet it there:https://gist.github.com/regit/1e591311fa3ba5cd0b8d73940348599a


https://gist.github.com/regit/1e591311fa3ba5cd0b8d73940348599a






6 Conclusion


Suricata eBPF bypass architecture


Test methodology

Test setupIntel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHzIntel Corporation 82599ES 10-Gigabit SFI/SFP+Live traffic:

Around 1Gbps to 2GbpsReal users so not reproducible

TestsOne hour long runDifferent stream depth valuesCollected Suricata statistics counters (JSON export)Graphs done via Timelion(https://www.elastic.co/blog/timelion-timeline)


https://www.elastic.co/blog/timelion-timeline

Results: stream bypass at 512kb


A few words on graphics

Tests at 512kbWe have on big flow that kill thebandwidthCapture get almost nullEven number of closed bypassedflows is low


Results







6 Conclusion


A Linux kernel feature

Run a eBPF code the earliest possiblein the driverin the cardbefore the regular kernel path

Act on dataDrop packet (eXtreme Drop Performance)Transmit to kernelRewrite and transmit packet to kernelRedirect to another interfaceCPU load balance


Implementation

Similar to eBPF filterSame logic for bypassOnly verdict logic is different

But annoying differenceeBPF code does the parsingNeed to bind to an interface


Suricata XDP architecture


AF_PACKET IPS mode

IPS and bypassCan’t drop packet at bypass stageNeed to forward from one iface to another


XDP and IPS mode: bypass

Use TX_REDIRECTDirect copy from interface to interface


Results

Direct NIC to NIC transferSkip all kernel taskWire speed copyIf eBPF code is fast enough

Obtained performanceTODO: Ask OISF marketing for some fake numbers to show







6 Conclusion


Conclusion

Suricata, eBPF and XDPAvailable in Suricata 4.1, need Linux 4.16Network card bypass for Netronome comingAF_XDP capture is now in Linux vanilla

More informationStamus Networks: https://www.stamus-networks.com/Septun II: https://github.com/pevma/SEPTun-Mark-II/Suricata doc: http://suricata.readthedocs.io/en/latest/capture-hardware/ebpf-xdp.html


https://www.stamus-networks.com/

https://github.com/pevma/SEPTun-Mark-II/

http://suricata.readthedocs.io/en/latest/capture-hardware/ebpf-xdp.html

http://suricata.readthedocs.io/en/latest/capture-hardware/ebpf-xdp.html

Questions ?

Thanks toJesper Dangaard BrouerAlexei StarovoitovDaniel Borkmann

Contact [email protected]: @regiteric

Want more fun ?Come to Suricata trainings: https://suricata-ids.org/training/

Suricon: https://suricon.net/


https://suricata-ids.org/training/

https://suricata-ids.org/training/

https://suricon.net/

suricata and xdp, performance with a s like security · kernel ask userspace for decision on...

Documents