black box testing methodology sans.ppt box testing... · why black box test? ! ... why black box...

© Copyright Red Tiger Security – Do not print or distribute without consent.

Black Box Testing Methodologies

Joe Cummins, PCIP, OPST Jonathan Pollet, CISSP, CAP, PCIP January 24, 2011 SANS SCADA Webinar, SCADA Summit Series 2011

welcome


Outline

� Why Black Box Test?

�  Layered approach

�  Black Box vs. White Box

�  Components of an Assessment

�  Process

�  Reports and metrics


…Why Black Box testing?

� Know what you are putting out on the network…

� How does a device respond to protocols it does not recognize?

� What happens when it gets a confusing message?

… are you sure?


Phased Approach to Device / Application Testing

�  Protocol �  RFC’s �  Proper communications

�  Software �  DOS, Overflow, Etc… �  Kernel

�  Firmware �  Assembler

�  Hardware �  Components �  Monitoring

OS

Firmware

Hardware

• Applications • Kernel

• Assembler


Layered Defence

6. Embedded Device

5. Communication Method

4. Servers / Workstations

3. DMZ

2. Infrastructure


Software / Middleware

� Exceptions �  Failures �  Null Pointers �  Access Violations

� Memory Corruption �  Buffer Overflow �  Stack Overflow


Hardware

� Components �  NIC (wired, wireless) �  Ports

� Monitoring �  CPU �  Temperature �  Cycles �  Processes �  Stack


Tools of the Trade


Manual Code Review

� Automated tools �  Highlights errors / changes �  Known common application faults �  Verification of Syntax

� Viewers �  Import / Export Source �  Render �  Analyze


Analysis Engine

� Core Fuzzing Process �  Reliance on the Tools and plugins to generate proper data

� Manual Code Review �  Line by line review

� Blended Analysis


Blended Analysis

� Device Testing Methodology

� Combination of both aspects �  Code review + Fuzzing = closer examination

� Benefits of both forms of Analysis


Anatomy of the Analysis

Model to Mayhem


White Box vs. Black Box Testing

Delivery

Application Implementation Protocol Specification

Function

Design Abstraction Dissection

Analysis

Code Review Input Testing

Testing

Verification Validation


Analysis Engine

Final Deliverable

Input Modules

Protocol Template

Target

Seed File Session

Assembler

Sessions

Collection Method

EKG

Outputs

Core Fuzzing Process


Input Generation Methods (Invalid)

Error Collection

Isolated Element

Invalid Data


Input Generation Methods (Valid)

Valid Output

Isolated Element

Valid Data


Device EKG / ECG

ICMP •  Echo •  Reply •  Config

SNMP •  Status •  Agent • Manager

TCP • HTTP (S) • SSH (22) • TELNET (23)


Device EKG / ECG

� ICMP �  ICMP Echo / Reply �  Dropped Config, Delayed Response, etc…

� TCP �  Active Session, keep-alive, timeouts �  HTTPS, SSH, Telnet,

� SNMP �  Monitoring �  Statistics


Comparison and Contrast

� What does an error look like? �  How do you work with this information? �  What can be determined about the program / device? �  Can this lead to cascading errors?

� What can you do with an error? �  POC? �  Weaponization / Exploit Development


Exploit Weaponization (Stages)

Staged Attack Binary

Exploit Payload

Vuln.

Code

Socket

Packaged Exploit

Exploit


Output Collection

� Comparison and contrast �  Characteristics of an error �  Scale of vulnerability

� “Weaponization” �  Malicious code �  Payloads �  Repeatable

� Hardware EKG �  Health of the device �  “State” of the device


Reports and Metrics

� Black box testing Report: �  Spreadsheet of tests and outputs, �  Tools used, �  Findings, �  Recommendations, �  Remediation steps,

� Include: �  Packet Captures (in pcap) for replay �  Screen captures �  Outputs for future analysis


Wrap-up

� Devices need to be tested �  Vendors continue to “push” product to market �  Consumers need to be aware of the hazards

� Small investment / Resilient Devices

� Testing is CRITICAL

� Does not need to be resource intensive �  Complex task, automated and facilitated �  Part of the internal Testbed


Contact info:

Jonathan Pollet, CISSP, CAP, PCIP Red Tiger Security - USA office: +1.877.387.7733 web: www.redtigersecurity.com [email protected] Check out our Industry Forum and sign up for RSS feed:

Forum: http://www.redtigersecurity.com/forum/

Joe Cummins, PCIP, OPST Founder, Principal Consultant Red Tiger Security - Canada office: +1.877.387.7733 web: www.redtigersecurity.ca [email protected]

25

black box testing methodology sans.ppt box testing... · why black box test? ! ... why black box...

Documents