rorschach plots and network performance analysis
DESCRIPTION
Presented @ BSidesDC 2013 Washington, DC, October 20, 2013 Measuring the performance of network protocols that require determinism can be difficult with the existing set of tools. Tools like Wireshark can give you the details of the protocols themselves and some general statistics about the packet streams, but they don’t easily show the full set of traffic for those streams. Visual tools like Etherape can show you the full set of traffic streams, but don’t give you any idea of the nuances of the performance represented in those traffic streams. While at the National Institute of Standards and Technology (NIST), I built a tool capable of analyzing and displaying the performance of network protocols. The first generation of the tool was called the Industrial Ethernet Network Performance (IENetP) test tool and the second generation of the tool is called the Factory Equipment Network Testing (FENT) framework. Both these tools are available on SourceForge and are public domain. I have since left NIST and the tools haven’t been picked up by anyone. The FENT framework is useful for analyzing the performance of any deterministic protocol and reporting certain performance characteristics. It was originally designed for EtherNet/IP (Ethernet / Industrial Protocol), Modbus, Profinet, and other industrial Ethernet-based protocols, but has proven to be useful for many other protocols as well. The most beneficial part of the software has actually been the graphical analysis, which in many cases resemble Rorschach plots due to the subtle performance problems that show up as strange patterns in the data. My presentation will describe the FENT framework, present the tool in its current state, and display some of the more interesting results. It will also be a plea for someone to take up the open-source development of this project and move it forward. My new position does not leave me with enough time to dedicate to the project, so the project has been dormant for the last few months. I’ve received complements on the project from many industrial partners in the past and they would like to see further development, but that means that someone else has to take on the task.TRANSCRIPT
BSidesDC 2013 1
Rorschach Plots and Network Performance
AnalysisJim Gilsinn
Kenexis Consulting Corporation
October 19-20, 2013
BSidesDC 2013 2
Rorschach?
October 19-20, 2013
BSidesDC 2013 3
“Rorschach” Plots
October 19-20, 2013
BSidesDC 2013 4
“Rorschach” Plots
October 19-20, 2013
BSidesDC 2013 5
“Rorschach” Plots
October 19-20, 2013
BSidesDC 2013 6
ICS Environment
October 19-20, 2013
BSidesDC 2013 7
ICS Environment
October 19-20, 2013
BSidesDC 2013 8
ICS Systems
October 19-20, 2013
BSidesDC 2013 9
What’s This All About?
• I used to work at NIST
• I left about a year ago
• I worked on ICS network performance metrics, tests, and tools
• The test tools I developed have been dormant since leaving
• The vendors I worked with while at NIST want to tool
• My new employer won’t support open-source development
• I’m here to beg for help!
October 19-20, 2013
BSidesDC 2013 10
Performance Testing Methodology:Performance Metrics
• Publish/subscribe or peer-to-peer communications
• Main performance metric: Cyclic frequency variability/jitter
• Real-time EtherNet/IP uses publish/subscribe• Requested/Accepted Packet Interval (RPI/API)• Measured Packet Interval (MPI)
October 19-20, 2013
BSidesDC 2013 11
Performance Testing Methodology:Performance Metrics
• Command/response or master/slave communications
• Main performance metric: Latency
• Large numbers of protocols use this• Most (All?) PC-based server/client protocols – HTTP(S), (S)FTP, etc.• Most industrial protocols – Modbus/TCP, Profinet, Ethercat, etc.
October 19-20, 2013
BSidesDC 2013 12
IENetP Test Tool
• Industrial Ethernet Network Performance (IENetP)
• http://sourceforge.net/projects/ienetp/
• Current Version = 1.1.2, Released 2011-02-11
• Software Features• Analyze existing Wireshark captures• Allows user to override default EtherNet/IP filter• Isolates individual traffic streams• Determine cyclic jitter of those streams• Generates HTML report• Generates time-space & histogram graphs• Graphs allow zooming
October 19-20, 2013
NIST Performance Test Tool
• Industrial Ethernet Network Performance (IENetP) Test Tool
• Factory Equipment Network Testing (FENT) Framework
BSidesDC 2013 14
FENT Framework
October 19-20, 2013
PersonalityModule
PersonalityModule
PersonalityModule
Universal Client Application API
Universal Client Application
TestingModule Analysis
EngineTestingModuleTesting
ModuleTestingModule
ReportingEngineAnalysis
Engine
SensorGateway
Internet
Ethernet
Fieldbus
BSidesDC 2013 15
FENT Features
• All Analysis Features From IENetP• Analyze Wireshark capture files• Build graphs and reports of results
• Added Features• True multi-protocol support• Real-time testing capability• Extensible framework
October 19-20, 2013
BSidesDC 2013 16
FENT Personality Modules
• Wrapper for Driver Application
• Implement a TCP-socket interface for UCA-API messaging
• Build a simple XML-based PM Descriptor file
• Features• Describes Wireshark
parameters• Allows any protocol to be used• Can be built/loaded at run-time
October 19-20, 2013
Protocol PM
UCA – API
PM Descriptor
Driver App
17
FENT Framework Run-Time1. Testing Module Protocol PM –
Grab protocol-specific Wireshark parameters via UCA-API
2. Testing Module Wireshark – Start capturing traffic
3. Testing Module Protocol PM – Command driver app to communicate with DUT
4. Testing Module Wireshark – Stop Capturing traffic, process capture file using desired protocol and user parameters, generate PSML file
5. Analysis Engine – Read PSML file, analyze packets for desired metrics
6. Reporting Engine – Report data to user
October 19-20, 2013 BSidesDC 2013
Protocol PM
UCA – API
UCA
TestingModule
AnalysisEngine
ReportingEngine
PM Descriptor
Wireshark
DUT
PSML File
DriverApp
BSidesDC 2013 18
FENT UCA-API Schema
October 19-20, 2013
BSidesDC 2013 19
FENT Framework
• Project Home:• http://sourceforge.net/projects/fent/
• What’s Available:• SVN repository & schema• FENT software
• Conduct real-time testing• Analyze results• Build graphs on-screen
• NIST SensorSim PM, IEEE 1451 PM• EtherNet/IP PlugFest “Gold Standard” Background Traffic
October 19-20, 2013
BSidesDC 2013 20
FENT Framework
• Known Problems & Issues• Doesn’t work with Wireshark 1.9+
• Tshark argument for getting fields changed
• Logic problems with using multi-protocol Wireshark headers• Software doesn’t use true database• Testing automation not integrated• No installer
October 19-20, 2013
BSidesDC 2013 21
FENT Demo
October 19-20, 2013
BSidesDC 2013 22
“Gold Standard” Background Traffic
October 19-20, 2013
BSidesDC 2013 23
“Gold Standard” Background Traffic
• What Is It?• A set of Wireshark captures, Linux scripts, and analysis results• Based on EtherNet/IP PlugFest performance testing requirements• High precision and accuracy Wireshark captures of PlugFest
performance background traffic• Linux scripts designed for use in BackTrack Linux (
http://www.backtrack-linux.org/)• Analysis results show validation for use in PlugFest performance
testing
• Where Can You Get It?• http://ienetp.sourceforge.net/EtherNet-IP_Testing.zip or• FENT SVN in Background_Traffic folder
October 19-20, 2013
PlugFest Performance Traffic
Traffic Type Rate (pps)
Baseline
Steady-State Managed
Steady-State Unmanaged
Burst Managed
Burst Unmanaged
ARP Request Broadcasts 180
Gratuitous ARP Broadcasts 180
DHCP Request Broadcasts 100
ICMP (ping) Request Broadcasts 100
NTP Multicasts 10
EtherNet/IP ListIdentity Request 10
EtherNet/IP Class 1 1800
ARP Burst Requests 240 pkts @ 4k Hz
BSidesDC 2013 25
“Gold Standard” Captures
• Built From Individual Traffic Streams• Each traffic stream generated and captured using NIST Ixia system (a
few microseconds jitter)• Assembled using editcap and mergecap scripts• Final captures are 60-seconds long
• Can’t just loop continuously• Longer test captures require rebuilding (not hard)
• Analyzed Using IENetP• Analysis results are included in package• Well within spec for PlugFest performance testing needs (<25% of
desired packet intervals)
October 19-20, 2013
BSidesDC 2013 26
Licensing?
• The project is Public Domain!!!
• There are NO LICENSING ISSUES!!!
October 19-20, 2013
BSidesDC 2013 27
What’s Next?
• Contact Me• Jim Gilsinn• 301-706-9985• [email protected]• Twitter – @JimGilsinn• LinkedIn – http://www.linkedin.com/in/jimgilsinn/
• Review the FENT SourceForge Project• http://sourceforge.net/projects/fent/
• Fork the Project
October 19-20, 2013