17 March 2016

Elbert Michael Ruiz,

Fred Wright, PhD,

Ronald Prado, Douglas Woods

Weapon Systems and Cyber Testing and Evaluation

2

Outline

Challenges/Impetus of Weapon Systems T&E

Methodology: Leveraging Systems Engineering

Lessons Learned and Observations

Needs and Technology Gaps

Impetus for Topic

Risk Management FrameworkSystem Development Life Cycle Activities

• Cyber Security Directive now applies to all DoD IT (including Platform IT)

• Confidentially, Integrity, Authorization, and non-repudiation

• Cybersecurity fully integrated into system lifecycles

• Cybersecurity T&E is conducted throughout the acquisition lifecycle

• Two phases:

• 1) Cooperative Vulnerability and Penetration Assessment,

• 2) Adversarial Assessment

• Includes all digital paths, not just Ethernet

IA Focused Cyber SecurityFocused

3

4

Cyber-Physical System Examples

Cyber-physical systems can also be self-contained (condensed System-of-Systems)

Commercial vehicles exhibit heterogeneous types of vulnerability vectors

Natural extension to apply concepts, techniques, and procedures to military Weapons Systems

Multi-stage attack sequences

Vehicle-to-VehicleCommunications

RSU-to-VehicleCommunications

Keyless Entry

Tire PressureSystem

InfotainmentUnit

Telematics

EngineControl

Unit

Transmission Control Unit

CAN Bus Controller

SecuritySystem

Anti-LockBrakes

On-Board Diagnostics

ClimateControl

http://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/

5

Weapons Systems as Cyber-Physical Systems

6

Outline

Challenges/Impetus of Weapon Systems T&E

Methodology: Leveraging Systems Engineering

Lessons Learned and Observations

Needs and Technology Gaps

Cybersecurity T&E Phases mapped to

Acquisition Life Cycle

Department of Defense

Cybersecurity Test and Evaluation GuidebookVersion 1.0July 1, 2015

Most high-level methodology descriptions call out the the actual assessment phase, but what exactly does it entail?

8

Starting Points - Assessment Methodology

CYBER & RF

Experts

Assessment Report

Attack Tree Analysis

• System Documentation• Block diagrams• Subsystem Detail• Interconnectivity

• Interviews• On-site Examination• Access to system• Risk Man. Framework

documentation

Subsystem Vulnerability Assessment

• Most likely ways in to affect mission

• Ability to Detect,Prevent, React, Recover from Cyber Attack

• Potential vulnerability to disruption, spoofing, malware

• Attack Surfaces• Network Nodes

• Propagation paths

Subsystem Identification

• Delphi Method• Subject Matter Experts

(e.g.)• Embedded SW

Reverse Engr (RE)• RF Protocol RE• Hardware RE

Vulnerability Analysis Deep

Dive

Develop Exploits

Demonstration/Red Team

Events

• Focus on likely paths• System Simulation• System Hardware• System SW• SME capabilities (SW Defined

Radios, fuzzers, spoofers, signal monitors)

• Difficult/Time consuming

• Malware almost exclusively 0-day

• Disrupt/Spoofing easier

• Validates ability of adversary

• Can assess and test mitigations

• Risks and Recommended Actions

T&E CapabilityNeeded

Common Denominator

9

Attack Tree Analysis (ATA)

Paradigm for performing hostile threat risk analysis using a rigorous tree-structured mathematical approach

Techniques first published and described in the early 1990’s

Based on Fault-Tree Analysis methodology developed in the 1960’s/70’s

ATA methods incorporate not only details of the system being defended, but also methods available to the attacker

Attack tree models excel at estimating the risk for situations where events happen infrequently or have never happened before

Attack Tree Example: Backdoor Scenario

10

11

RF links/comms

GPS

Supply chain (hardware and software)

Maintenance interfaces

Reprogramming interfaces

IT enterprise connections

Command and control systems

Mission planning

Training systems

Common Attack Vectors for Weapon Systems

System Cross-Functional Cyber Analysis

12

RF System Vectors

SoftwareVectors

HardwareVectors

Network Vectors

Comms/GPS/etcMaintenance Ports/Anti-Tamper

Auto-pilot/HUDs/Collision Avoidance

TCP-IP/ System Busses

Vulnerability Assessment Methodology for Embedded Systems

13

Operational Assessment

Attack Surface Enumeration /

Risk Assessment

Message Reverse

Engeineering

MessageGeneration

Experiments / Technique

Refinement

Download Code

Initial Disassembly/ Static Analysis

Dynamic Analysis

HardwareReverse

Engineering

Instrument- ation

RF Link Insertion / Propagation

Analysis

Lab or Field Test DemonstrationsSoftware Analysis

Communications Analysis

Software Assessment

Hardware Assessment

Communications Assessment

Exploit Development / Experiments /

Technique Refinement

SDR Development

Phase I Phase II Phase III

Our full 3-phase approach for vulnerability analysis and exploitation

SDR=SW Defined Radio

14

Outline

Challenges/Impetus of Weapon Systems T&E

Methodology: Leveraging Systems Engineering

Lessons Learned and Observations

Needs and Technology Gaps

15

Develop the assessment team Avionics and Electronic Warfare (EW) platform cyber

assessment expertise in developmental stages

Cross-section of embedded system reverse engineering skill-sets

Technically proficient in RF and digital protocol analysis, firmware reverse engineering, embedded vulnerability discovery and exploit

Vulnerability discovery and exploit important step Improves confidence of current and future assessment scoring

Confidence in assessment scoring builds over time (multiple platform assessments)

Commonalities in cross-platform subsystems inform – likely vulnerabilities

Best Practices/ Lessons Learned

16

Start assessments early

System research is inherently time-consuming.

Identification and demonstration of vulnerabilities even more time-consuming

Provide access to the necessary platform experts to the assessment team

Pilots, trainers, maintenance crew, sub-system SMEs

Access to detailed system/subsystem information critical

Message formats, timing, and protocols between platform subsystems

Include all digital paths (1553, serial, Ethernet, RF)

Board schematics and firmware for each relevant subsystem

Best Practices/ Lessons Learned (Continued)

17

Access to System System Integration Labs

Hardware in the Loop Labs

Off-site System/subsystem analysis – Subsystems available to assessors at their facilities

Plan multiple trips for assessors

Reuse past analysis and assessment results and patterns Device evaluations should feed “platform” evaluations

Feed back into hardware and software standards/implementation guides, security patterns and approaches Security is built-in to systems engineering

Best Practices/ Lessons Learned (Continued)

18

Outline

Challenges/Impetus of Weapon Systems T&E

Methodology: Leveraging Systems Engineering

Lessons Learned and Observations

Needs and Technology Gaps

19

Cross-Functional vector tracking/test control (https://kepler-project.org/ & http://ptolemy.eecs.berkeley.edu/ )

Smart Fuzzers

Intelligent protocol/stack reverse engineering tools

Open standards with reference architectures and controls for security (e.g., Future Airborne Capability Environment (FACE™))

Persistent laboratories with these capabilities and weapon systems “stacks” to verify implementations in DT

Including simulations of controlled processes (real-time but not necessarily high-fidelity)

Gaps and Needs

Smart Fuzzer and Automated Protocol Reverse Engineering Concepts

Example protocol analyzer structure using machine learning (e.g., deep learning techniques)

Support for thousands of simultaneous tests-Reuse/share environment assets as appropriate

Analysts can make informed decisions on which bugs to target

21

Assessments have been successfully executed and lessons learned are available

Time and cost must be balanced with “depth” of assessment

Need: Integration with systems engineering/design to ensure better security

Need: automation to reduce time and cost

Summary