kernel recipes 2015: kernel packet capture technologies
TRANSCRIPT
![Page 1: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/1.jpg)
Kernel packet capture technologies
Éric Leblond
Stamus Networks
October 1, 2015
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 1 / 54
![Page 2: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/2.jpg)
1 Introduction
2 Why capture
3 Libcap and raw socket
4 AF_PACKET
5 PF_RING
6 AF_PACKET goes multi*
7 Netmap
8 Latest AF_PACKET evolution
9 ++zero copy
10 Conclusion
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 2 / 54
![Page 3: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/3.jpg)
Éric Leblond
Co-founder of Stamus NetworksCompany providing network probe based on SuricataFocusing on bringing you the best of Suricata IDS technology
Open source hackerSuricata core developerNetfilter core team member
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 4 / 54
![Page 4: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/4.jpg)
Raw socket: definition
A raw socket is an internet socket that allows direct sending and receiving of InternetProtocol packets without any protocol-specific transport layer formatting.
Wikipedia
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 6 / 54
![Page 5: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/5.jpg)
"The End of the Internet"
[raw socket ...] spells catastrophe for the integrity of the Internet.
Steve Gibson in 2001
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 7 / 54
![Page 6: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/6.jpg)
"The End of the Internet"
Talking about introduction of raw socket in MS WindowsAllow users to write any packetsCould be used to abuse protocol and [poorly implemented] OS
More info at http://www.informit.com/articles/article.aspx?p=27289
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 8 / 54
![Page 7: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/7.jpg)
Raw socket: usage
Send and receiveSend low level message: icmp, igmpImplement new protocol in userspace
SniffingCapture trafficPromiscuous modeUse by network monitoring tools
Debugging tools: tcpdump, wiresharkMonitoring tools: iptraf, ntop, NSAIntrusion detection systems: snort, bro, suricata
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 9 / 54
![Page 8: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/8.jpg)
Network Intrusion Detection System: definition
An intrusion detection system (IDS) is a device or software application that monitorsnetwork or system activities for malicious activities or policy violations and producesreports to a management station.
Wikipedia
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 10 / 54
![Page 9: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/9.jpg)
Network Intrusion Detection System: challenge
IDS detection rule
Some dataComplexity of rule
Work on recontructed streamProtocol field analysisPattern recognition on ungzipped content (http_server_body)
Got around 15000 rules in standard rulesetNeed to inspect 10Gbps of trafic or more
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 11 / 54
![Page 10: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/10.jpg)
Suricata: Open source & multi threaded IDS
IDS and IPS engineGet it here: http://www.suricata-ids.orgProject started in 2008Open Source (GPLv2)Funded by consortium members (and originaly USgovernment)Run by Open Information Security Foundation (OISF)More information about OISF athttp://www.oisf.net/
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 12 / 54
![Page 11: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/11.jpg)
Suricata Features
High performance, scalable through multi threadingProtocol identificationFile identification, extraction, on the fly MD5 calculationTLS handshake analysis, detect/prevent things like DiginotarHardware acceleration support:Useful logging like HTTP request log, TLS certificate log, DNS loggingLua scripting for detection
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 13 / 54
![Page 12: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/12.jpg)
libpcap
Multi OS abstraction for packet captureAll *nix, WindowsMulti layer: Network, USB, . . .
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 15 / 54
![Page 13: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/13.jpg)
Raw socket: the initial implementation
A dedicated socket type
#include <sys / socket . h>#include < n e t i n e t / i n . h>raw_socket = socket ( AF_INET , SOCK_RAW, i n t p ro toco l ) ;
Straight socket modeGet packet per packet via recvmsgOptional ioctl
Get timestamp
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 16 / 54
![Page 14: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/14.jpg)
Memories of another time
"640 K ought to be enough for anybody." Memory contraint designNo preallocationOn demand only
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 17 / 54
![Page 15: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/15.jpg)
Disclaimer
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 18 / 54
![Page 16: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/16.jpg)
IDS design
Monoprocess
No Performance for you, go home now.
Marty Roesch about multithread and network data processing, 2010
Suricata architecture
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 19 / 54
![Page 17: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/17.jpg)
NAPI (2001-200?)
Reducing interrupts usageInterrupts tempest at high packet rateAll CPU time is sued to handle the interruptsNIC driver needs to be updated
No direct change for packet captureChange internal to device driverDirect performance impact on packet capture
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 21 / 54
![Page 18: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/18.jpg)
NAPI performance
Table extracted from luca.ntop.org/Ring.pdf
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 22 / 54
![Page 19: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/19.jpg)
Problem of the socket mode
Internal pathData in card bufferData copied to skbData copied to socketData read and copied by userspace
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 23 / 54
![Page 20: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/20.jpg)
Memory map approach
Sharing is the solutionKernel expose some memoryUserspace access memory directlySpare a message sending for every packets
mmap internal pathData in card bufferData copied to skbData copied to ring bufferUserspace access data via pointer in ring buffer
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 24 / 54
![Page 21: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/21.jpg)
TPACKET_V2
setupsocket(): creation of the capture socketsetsockopt(): allocation of the circular buffer (ring) via PACKET_RX_RING optionmmap(): mapping of the allocated buffer to the user process
capturepoll(): to wait for incoming packets
shutdownclose(): destruction of the capture socket and deallocation of all associated resources.
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 25 / 54
![Page 22: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/22.jpg)
Memory organization
Ascii artblock #1 block #2
+---------+---------+ +---------+---------+| frame 1 | frame 2 | | frame 3 | frame 4 |+---------+---------+ +---------+---------+
block #3 block #4+---------+---------+ +---------+---------+| frame 5 | frame 6 | | frame 7 | frame 8 |+---------+---------+ +---------+---------+
ComponentsFrame contains a datagram dataBlocks are physically contiguous region of memory
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 26 / 54
![Page 23: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/23.jpg)
Performance
Graph extracted from luca.ntop.org/Ring.pdf
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 27 / 54
![Page 24: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/24.jpg)
Suricata architecture
MMAP optionSupport of TPACKET_V2Zero copy mode
Implied changesAccess data via pointer to ring buffer cellRelease data callback
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 28 / 54
![Page 25: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/25.jpg)
PF_RING original design (2004)
Architecturering designmmapcapture only interface
skip kernel pathput in ring buffer and discard
user access the ring buffer
ProjectProject started by Luca DeriAvailable as separate sources
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 30 / 54
![Page 26: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/26.jpg)
PF_RING performance
Show real improvement on small size packetsPre optimisation resultBetter result in following version due to a better poll handling
Table extracted from luca.ntop.org/Ring.pdf
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 31 / 54
![Page 27: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/27.jpg)
PF_RING going multicore (around 2008?)
Sharing the loadEach core has a finite bandwidth capability
Multicore CPU were introduced in 2006Sharing load become common
Previously separate hardware was used to split the network load
Straight forward solutionAllow multiple sockets to be attached to one interfaceLoad balance over the attached sockets
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 32 / 54
![Page 28: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/28.jpg)
Suricata autofp multi reader
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 33 / 54
![Page 29: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/29.jpg)
PF_RING code
Build system and sourcesCustom build systemNo autotools or cmakeInclude patched drivers
SVN stats
g i t log −−format=format : "%s " | s o r t | uniq −c | s o r t −n | t a i l −n1015 Minor change20 f i x20 minor changes22 l i b re f resh30 L i b r a r y re f resh43 minor change67 minor f i x
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 34 / 54
![Page 30: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/30.jpg)
David Miller in da place
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 36 / 54
![Page 31: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/31.jpg)
AF_PACKET load balancing (2011)
Multiple sockets on same interfaceKernel does load balancingMultiple algorithms
LB algorithmRound-robinFlow: all packets of a given flow are send to the same socketCPU: all packets treated in kernel by a CPU are send to the same socket
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 37 / 54
![Page 32: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/32.jpg)
AF_PACKET CPU Load balancing
RSS queuesMultiqueue NIC have multiple TX RXData can be split in multiple queues
Programmed by userFlow load balanced
RSS queues load balancingNIC does load balancing using hash functionCPU affinity is set to ensure we keep the cache line
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 38 / 54
![Page 33: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/33.jpg)
Suricata workers mode
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 39 / 54
![Page 34: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/34.jpg)
tpacket_v3 (2011)
The problemCell are fixed sizeSize is the one of biggest packet (MTU)Small packets use same memory as big one
Variable size cellsRing bufferUpdate memory mapping to enable variable sizesUse a get pointer to next cell approach
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 40 / 54
![Page 35: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/35.jpg)
Netmap (2012)
Similar approach than PF_RINGskip kernel pathput in ring buffer and discard
User access the ring bufferPaired with network card ring
More info http://queue.acm.org/detail.cfm?id=2103536
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 42 / 54
![Page 36: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/36.jpg)
Performances
Table by Luigi Rizzo
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 43 / 54
![Page 37: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/37.jpg)
AF_PACKET rollover option (2013)
Single intensive flowLoad balancing is flow basedOne intensive flow saturate core capacityLoad needs to be shared
PrincipleMove to next ring when ring is fullAs a load balancing modeAs a fallback method
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 45 / 54
![Page 38: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/38.jpg)
Rollover and suricata (1/2)
Graph by Victor Julien
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 46 / 54
![Page 39: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/39.jpg)
Rollover and suricata (2/2)
A TCP streaming issueRollover activation lead to out of order packetsFool TCP stream reconstruction by suricataResult in invalid streams
Possible solutionEvolve autofop multicaptureDecode and dispatch packets
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 47 / 54
![Page 40: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/40.jpg)
DPDK (2012-)
Data Plane Development Kitset of libraries and driverdesign for fast packet processingimpact on software architecture
Architecturemulticore frameworkhuge page memoryring bufferspoll-mode drivers
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 48 / 54
![Page 41: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/41.jpg)
Suricata workers mode limit
Packet treatment can be really longInvolve I/O on disk or networkHuge computation like regular expression
Ring buffers are limited in sizeA slow packet can block a whole bufferSuricata need to dequeue faster
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 50 / 54
![Page 42: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/42.jpg)
Need to evolve Suricata architecture
Switch to asynchronousRelease ring buffer elements as fast as possibleBuffer in userspace
An enhanced autofp approach?Fast decodeCopy data to packet pool of detect threadWith a fast decisionRelease data
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 51 / 54
![Page 43: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/43.jpg)
Conclusion (1/2)
A small subject and a huge evolutionHas follow evolution of hardware architectureAlways need to deal with more speed
10Gbps is common100Gbps is in sight
Multiple technologiesVanilla kernel propose some solutionsPatching may be required to do more
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 53 / 54
![Page 44: Kernel Recipes 2015: Kernel packet capture technologies](https://reader031.vdocuments.mx/reader031/viewer/2022020119/5872ddab1a28ab74188b78c7/html5/thumbnails/44.jpg)
Conclusion (2/2)
Do you have questions ?
Contact meMail: [email protected]: @Regiteric
More informationSuricata: http://www.suricata-ids.orgPF_RING: http://www.ntop.org/products/packet-capture/pf_ring/netmap: http://info.iet.unipi.it/~luigi/netmap/dpdk: http://dpdk.org/
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 54 / 54