cyber-physical v&v challenges for the evaluation of state ... · cyber-physical v&v...

Copyright 2016, Lockheed Martin Corporation. All rights reserved.

Cyber-Physical V&V Challenges for the Evaluation of State of the Art Model CheckersResearch in Quantum Enabled V&V Technology

July 12-14, 2016

Chris ElliottFlight Controls / Quantum Computing

2Copyright 2016, Lockheed Martin Corporation. All rights reserved.

Overview

I. Quantum Enabled V&V Overview

II. Overview of 10 V&V Challenge Problems

III. End to End Analysis Example

IV. Summary


What is it? • QVTrace*: This technology is a method for Software Verification &

Validation using Quantum Computer Assisted Formal Methods.

RequirementsAnd Implementation

(Software Code)

Classical Computation

D-Wave Adiabatic Quantum Computer

Defects (Bugs)Req/Code

InconsistencyReport to Designer

Quantum V&V

*Product Developed by QRA Inc.

• Target Users are System/Software Design Teams interested in:- Reducing development costs - Improving final product quality

Who will use it?

Quantum Enabled V&V


LM QA Solves a Quadratic Unconstrained Binary Optimization Problem

D-Wave Adiabatic Quantum ComputerCurrent State-of-the-Art, DW-2X: 1152q Washington


Quantum Superposition, Entanglement Enable Unique Optimization

Quantum Optimizationwith Superconducting Qubits


Early 2010LM ID’s

Quantum asKey Tech

Nov 2010Early Access

To QC

Mar 2011USC/ISI/LMTeam for QC Center

Jan 2012USC-LM

QC Operational

Mar 2013QC Upgrade

March 2016QC Upgrade

128 q DW1 “Rainier”512 q DW2 “Vesuvius”

1152 q DW2X “Washington”

QE-V&V Timeline


Overview of Challenge Problems• LM Aero Developed Set of 10 V&V Challenge Problems

• Goal: - Foster Collaboration in S5 Community (Ponder, Present, Publish)- Evaluate & Improve State-of-the-Art Formal Methods Toolsets

• Each Example in Package Includes:- Simulink Model Built in Matlab® R2012B- Parameters, if any, for Simulating Model (.mat)- Documentation Containing Description and Requirements

• Difficult due to Transcendental Functions, Nonlinearities and Discontinuous Math, Vectors, Matrices, States

Challenges Built with Commonly Used Blocks


Overview of Challenge Problems1. Triplex Signal Monitor2. Finite State Machine3. Tustin Integrator4. Control Loop Regulators5. Nonlinear Guidance Algorithm6. Feedforward Cascade Connectivity Neural Network7. Abstraction of a Control Allocator (Effector Blender)8. 6DOF with DeHavilland Beaver Autopilot*9. System Safety Monitor10. Euler Transformation

Flight Control and Vehicle Management System Inspired Problems


1. Triplex Signal Monitor

Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p

Sensor A

Sensor B

Sensor C

Airborne Redundancy Management

OnlineMonitoring


2. Finite State Machine


Discrete Interwoven Modes in Integrated Cyber-Physical System

Flight ControlEmbedded System

IntegratedSensor


3. Tustin Integrator


Fundamental Modeling and Simulation Component

NumericalIntegration


Kp

Gain

Kd

Gain1

Ki

Gain3

1s

Integrator

s

s+1

Transfer Fcn

4. Control Loop Regulators


CommandAuthority?Feedback

ErrorSynthesis

PIDArchitecture

Attributes of Multi-Axis Control Law of Output Commands


5. Nonlinear Guidance Algorithm


3D Vector Mathematics for Outer Loop Intercept Guidance

Aim Point Validity?


5. …Nonlinear Guidance Algorithm

Block TypesFor NL Guidance

Recent Focus on Import of Common Algorithmic Operators (Primitives)


6. Neural Network


2x10x10x1 Feedforward Cascade Connectivity NN

Output Features?

Inputs Layer 1 Layer2 Output

2-y

1-x

Network Topology

Inpu

ts

0.0 - 0.293130.29313 - 0.586260.58626 - 0.879380.87938 - 1.17251.1725 - 1.46561.4656 - 1.75881.7588 - 2.0519

PositiveNegative

-0.2

2

0

0.2

0.4

1

z

0.6

0.8

2

1

y

1.50 10.5

x

0-1 -0.5-1

-1.5-2 -2

Truth Model


… and More


Cyber-Physical V&V Challenge ProblemsLM Aeronautics Quantum Information Science Research Team 2015

Copyright © 2015 Lockheed Martin Corporation

AD

ID

AP Eng

HDG Mode

ALT Mode

HDG Ref

Turn Knob

ALT Ref

Pitch Wheel

Aileron Cmd

Elevator Cmd

Rudder Cmd

Autopilot

AC Bus

ID

AD

Sensors

Aileron

Elevator

Rudder

Flap

Throttle

Rudder Trim

Controls

Signal Conditioning

trim_flap

Constant

trim_throttle

Constant1

trim_rudder

Constant2trim_hdgref

Constant3

trim_turnknob

Constant4

trim_altref

Constant5

trim_pitchwheel

Constant6

DeHavilland Beaver Airframe

EnvBus

Environment

DeHavilland Beaver model originally based on work created by

Marc Rauw for Delft University of Technology, http://www.dutchroll.com

and subsequently modified by the Mathworkshttp://www.mathworks.com/matlabcentral/fileexchange/

FLIGHT CONTROL DemonstrationAuthor: elliocmModel Version: 1.80Date: 21-Sep-2015 15:23:12

boolean

Data Type Conversion1

boolean


boolean


1

Constant7APeng

HDGmode

HDGref

TurnKnob



End to End Analysis (Tustin)Cyber-Physical V&V Challenge Problems

LM Aeronautics Quantum Information Science Research Team 2015Copyright © 2015 Lockheed Martin Corporation

1

yout

2reset

1

xin

3T

xin

T

TL

BL

reset

ic

yout

TustinIntegrator

(Limited, Resettable, States)

4ic

5

TL

6

BL

cmd

Definitions:• Normal operation: the integrator is not in reset mode, and the

output is within the specified limits (TL and BL).• ypv: prior yout value• xinpv: prior xin input value• SP: Saturation Point

Input Signal to Be Integrated

Discrete Time Step

Top Limit

Bottom Limit

Boolean Reset

Initial Condition Upon Reset

Output Signal

Documentation Provides ICD, Definitions, and Requirements



End to End Analysis (Tustin)

TUSTIN INTEGRATOR (LIMITED, RESETTABLE, STATES)

Product

~=

Switch

upu

loy

SaturationDynamic

1

yout

.5

Gain

5reset

1xin

2T

4BL

3TL

6ic

z1

Unit Delay

z1

Unit Delay1

[TL]

Goto

TL

BL

TLc

BLc

bounds

[BL]

Goto1

[TL]

From

[BL]

From1




~=

Switch1

<

Relational Operator

1TL

2BL

1TLc

~=

Switch2

2BLc


Requirements: 1. When Reset is True and the Initial Condition (ic) is bounded by the provided Top and

Bottom Limits (BL <= ic <= TL), the Output (yout) shall equal the Initial Condition (ic).2. The Output (yout) shall be bounded by the provided Top and Bottom limits (TL and BL)3. When in normal operation, the output shall be the result of the equation, yout = T/2*(xin

+ xinpv)+ ypv4. The Output of this function shall approximate the integration of the value of the input

signal over time within a specified tolerance, defined in subtests below:a. After 10 seconds of Computation at an execution frequency of 10 hz, the Output

should equal 10 within a +/- 0.1 tolerance, for a Constant Input (xin = 1.0), and the sample delta time T = 0.1 seconds when in normal mode of operation.

b. Over a 10 second computational duration at an execution frequency of 10 hz, the Output should equal the sine of time t, sin(t), where time is defined as a vector from 0 to 10 by increments of 0.1 seconds within a +/- 0.1 tolerance for an input equal to the cosine of time t, cos(t), with the sample delta time T = 0.1 seconds when in normal mode of operation.


Requirements Properties (Tests) is At Least Half the Challenge









Detailed Formal Property Derivation:# 1. When Reset is True and the Initial Condition (ic) is# bounded by the provided Top and Bottom Limits (BL<=ic<=TL),# the Output (yout) shall equal the Initial Condition (ic).# If the Initial Condition is not bound by the Limits # during a Reset, the Output shall equal the saturation # point (nominally with TL>=BL, ic>=TL impl SP==TL and ic<=BL implSP==BL. # Off-nominally with TL<BL, ic, ic>=BL impl SP==BL and ic<=TL implSP==TL.((reset and ic<=TL and ic>=BL) impl yout == ic); #1a ((reset and ic>=TL and ic>=BL and TL>=BL) impl yout == TL); #1b ((reset and ic<=BL and ic>=BL and TL>=BL) impl yout == BL); #1c((reset and ic>=BL and ic<=TL and TL<BL) impl yout == BL); #1d((reset and ic<=TL and ic>=BL and TL<BL) impl yout == TL); #1e



Requirements Properties (Tests) is At Least Half the Challenge







Detailed Formal Property Derivation:# Over a 10 second computational duration at an execution frequency of 10 hz, the Output should equal the sine of time t, sin(t), where time is defined as a vector from 0 to 10 by increments of 0.1 seconds within a +/- 0.1 tolerance for an input equal to the cosine of time t, cos(t), with the sample delta time T = 0.1 seconds when in normal mode of operation(xin{0}==1 and xin{1}==0.995 and … xin{100}==-0.83907 and T{all}==0.1 and reset{never} and (TL{all}>=BL{all}) and (yout{all}>BL{all}) and (yout{all}<TL{all})) impl (abs(yout{0}-0)<=0.1 and abs(yout{1}-0.099833)<=0.1 … and abs(yout{98}--0.36648)<=0.1 and abs(yout{99}--0.45754)<=0.1 and abs(yout{100}--0.54402)<=0.1);

Analytic vs Numerical

|Tustin Error| < .05

10 s

10 s



End to End Analysis (Triplex)

Signal A

Signal B

Signal C

Threshold Level

Persistence Limit (Duration Trigger)

Fault Code

Given These Conditions, Prove the Correct Fault Report

FC: 0-nofail, 1-branchC, 2-branchB, 4-branchA# detailed formal property(abs(ia{all}-ib{all})>Tlevel{all} or abs(ia{all}-ic{all})>Tlevel{all} and PC>PClimit and PClimit{all}==1 and Tlevel{all}==1) impl (FC{3}==4);

26

0 0.5 1 1.5 2 2.5 30

5000

10000

t [sec]

inpu

ts

iaibic

0 0.5 1 1.5 2 2.5 3-1

-0.5

0

0.5

1

t [sec]

FC

Counter Example Data As a Test Harness to Model

27

0 1 2 30

5000

10000

t [sec]

inpu

ts

iaibic

0 1 2 3-1

-0.5

0

0.5

1

t [sec]

FC

0 1 2 30

1

2

3

4

t [sec]

inpu

ts

|ia-ib||ia-ic||ib-ic|

This is a ValidDefect DiscoveredBy QVTrace v0.9.1

Closer Inspection Yields a Problem

28

0 1 2 3 4 5 6 7 8 9 10-4

-2

0

2

t [sec]

inpu

ts

iaibic

0 1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

t [sec]

FC

FC: 0-nofail, 1-branchC, 2-branchB, 4-branchA

Nominal Behavior

29

0 1 2 3 4 5 6 7 8 9 10-4

-2

0

2

t [sec]

inpu

ts

iaibic

0 1 2 3 4 5 6 7 8 9 10-1

-0.5

0

0.5

1

t [sec]

FC

FC: 0-nofail, 1-branchC, 2-branchB, 4-branchA

Faulty Behavior


Summary and Path Forward• Round 1 V&V Challenge Problems In Use to Develop Novel QE-V&V

• Requirements Formalization is Difficult Alone and Reduces Defects - Requirements Properties (Tests) is At Least Half the Challenge- Beneficial to Front Load Design Process with Formalization- Need Near if Not Equivalent “Primitives” Capability in Properties- Interested in Deploying Challenges Requirements to SPeAR

• Goals:- Publish Results on Current Round of Challenges - Round 2 V&V Challenge Problems To Increase Complexity Further - Transition Formal Methods Analysis Process/Tools to Programs- Interested? Contact: Chris Elliott,

[email protected], 817-935-3054

Thank You

mailto:[email protected]


Dr. Edward H. “Ned” AllenChief Scientist and LM Senior FellowLockheed Martin Corporation

Mr. Greg TallantProgram Manager and LM Fellow

Lockheed Martin AeronauticsSkunk Works

Biography slide

Chris ElliottQuantum Apps Team


Mr. Peter StanfillQuantum Apps TeamLockheed Martin AeronauticsSkunk Works

Dr. Kristen PudenzQuantum Apps Team


cyber-physical v&v challenges for the evaluation of state ... · cyber-physical v&v...

Documents