norman vincent peale, - ase...
TRANSCRIPT
– Norman Vincent Peale, The Power of Posi,ve Thinking
“Do not build up obstacles in your imagina:on.”
Pop Quiz What Do These Things Have in Common?
✓ They are governed by stochas:c phenomena
✓We have a reasonable understanding of their causes
✓We base our understanding on past observa:ons - At various levels of abstrac:on
(Simple) Answer: They are events to which
experts assign a probability based on models
Software is like this too!
Certainty inSoKware Engineering
“My program is correct.” “The specifica,on is sa,sfied.”
A simplistic viewpoint, which permeates most of our models, techniques and tools
Example Model Checking
! ¬p→◊q( )∧"( )
Model
Checker
✓✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
Example Model Checking
! ¬p→◊q( )∧"( )
Model
Checker
✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
Why Apply aProbabilis:c Viewpoint?
✓ Perfect and complete requirements are improbable
✓ Execu:on and tes:ng are akin to sampling … and we use tes:ng to increase confidence!
✓ The behavior of the execu:on environment is
random and unpredictable
✓ Frequency of execu:on failures is (hopefully) low
There are many random phenomena and “shades of grey”
in software engineering
But our models, techniques and tools rarely capture this
NATO Conference on SERome, 1969
h[p://homepages.cs.ncl.ac.uk/brian.randell/NATO/N1969/DIJKSTRA.html
– Edsger W. Dijkstra
(on more than one occasion!)
“Tes:ng shows the presence,
not the absence of bugs.”
Probabili:es at Garmisch, 1968 John Nash, IBM Hursley
h[p://homepages.cs.ncl.ac.uk/brian.randell/NATO/N1968/GROUP1.html
Naur & Randell, SoDware Engineering: Report on a Conference sponsored by the NATO Science CommiLee, Garmisch, Germany, 7th to 11th October 1968, January 1969.
Some Previous Efforts with
Probabilis:c Approaches• Performance Engineering (many)
• Cleanroom SoKware Engineering (Mills)
• Opera:onal Profiles and SoKware Reliability Engineering (Musa, …)
• Quan:ta:ve Goal Reasoning in KAOS (Lamsweerde, Le:er)
• Sta:s:cal Debugging (Harrold, Orso, Liblit, …)
• Probabilis:c Programming & Analysis (Poole, Pfeffer, Dwyer, Visser, …)
• Probabilis:c and Sta:s:cal Model Checking (many)
Probabilis:c Model Checking
! ¬p→◊q( )∧"( )
Model
Checker
✓✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
P≥0.95 [ ]
0.4
0.6
Probabilis:c
Probabilis:c
Probabilis:c Model Checking
! ¬p→◊q( )∧"( )
Model
Checker
✓✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
P=? [ ]
0.4
0.6
Quan:ta:ve Results
0.9732Probabilis:c
Probabilis:c
Example The Zeroconf Protocol
s1 s0 s2 s3
q
1
1
{ok} {error}
{start} s4
s5
s6
s7
s8
1
1-q
1-p
1-p
1-p 1-p
p p p
p
1
Pr(unsuccessful message probe)Pr(new address in use)
from the PRISM group(Kwiatkowska et al.)
P≤0.05 [ true U error ]
0.1 0.9
0.5
0.5
0.10.10.1
0.9
0.9
0.9
Some Previous Efforts with
Probabilis:c Approaches• Cleanroom SoKware Engineering
• Opera:onal Profiles & SoKware Reliability Engineering
• Quan:ta:ve Goal/Requirements Reasoning in KAOS
• Performance Engineering
• Sta:s:cal Debugging
• Probabilis:c Programming and Analysis
• Probabilis:c Model Checking
We lack an holistic approach
for the whole software development lifecycle
Challenges in Taking a
Probabilis:c Viewpoint
1. Some Things Are Certain, Or Should Be
2. Educa:on and Training
3. Popula:on Sizes and Sample Sizes
4. Determining the Probabili:es
5. Pinpoin:ng the Root Cause of Uncertainty
Challenges Some Things Are Certain, Or Should Be
Need to mix probabilistic and non-probabilistic approaches
Challenges Determining the Probabili:es
! ¬p→◊q( )∧"( )
Model
Checker
✓
State Machine
Model
Temporal Property
ResultsSystem
Requirements
P≥0.95 [ ]
0.4
0.6
Quan:ta:ve Results
0.9732Probabilis:c
Probabilis:c
Challenges Determining the Probabili:es
! ¬p→◊q( )∧"( )
Model
Checker
✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
P≥0.95 [ ]
Quan:ta:ve Results
Probabilis:c
Probabilis:c
0.41
0.59
0.6211
Example The Zeroconf Protocol Revisited
s1 s0 s2 s3
q
1
1
{ok} {error}
{start} s4
s5
s6
s7
s8
1
1-q
1-p
1-p
1-p 1-p
p p p
p
1
from the PRISM group(Kwiatkowska et al.)
The packet-loss rate is determined by an
empirically es,mated probability distribu,on
Pr(packet loss)
0.1 0.9
0.5
0.5
0.10.10.1
0.9
0.9
0.9
Perturbed Probabilis:c Systems (Current Research)
• Discrete-Time Markov Chains (DTMCs) • “Small” perturba:ons of probability parameters
• Reachability proper:es P≤p[ ] • DRA proper:es
• Linear, quadra:c bounds on verifica:on impact
see papers at ICFEM 2013, ICSE 2014, CONCUR 2014,ATVA 2014, FASE 2016, ICSE 2016, IEEE TSE 2016
• Markov Decision Processes (MDPs)
• Con:nuous-Time Markov Chains (CTMCs)
S? U S!
Asympto:cPerturba:on Bounds
• Perturba:on Func:on
where A is the transi:on probability sub-matrix for S?
and b is the vector of one-step probabili:es from S? to S!
• Condi:on Number
• Predicted varia:on to probabilis:c verifica:onresult p due to perturba:on Δ:
ρ x( ) = ι? i A x i ib x( )− Ai ib( )( )i=0
∞
∑
κ = limδ→0
sup ρ(x − r)δ
: x − r ≤ δ ,δ > 0⎧⎨⎩
⎫⎬⎭
p̂ = p ±κΔ
Case Study Results Noisy Zeroconf (35000 Hosts, PRISM)
p Actual Collision Probability
Predicted Collision Probability
0.095 -19.8% -21.5%
0.096 -16.9% -17.2%
0.097 -12.3% -12.9%
0.098 -8.33% -8.61%
0.099 -4.23% -4.30%
0.100 1.8567 ✕ 10-4 —0.101 +4.38% +4.30%
0.102 +8.91% +8.61%
0.103 +13.6% +12.9%
0.104 +18.4% +17.2%
0.105 +23.4% +21.5%
Challenges Pinpoin:ng the Root Cause of Uncertainty
“There are known knowns; there are things we know we know. We also know
there are known unknowns; that is to say, we know there are some things we
do not know. But there are also unknown unknowns – the ones we don’t
know we don’t know.”
— Donald Rumsfeld
The Changing Nature ofSoKware Engineering
✓ Autonomous Vehicles
✓ Cyber Physical Systems
✓ Internet of Things
see
Deep Learning and Understandability versusSoDware Engineering and Verifica,on
by Peter Norvig, Director of Research at Google
h[p://www.youtube.com/watch?v=X769cyzBNVw
Example Affec:ve Compu:ng
When is an incorrect emo,on
classifica,on a bug, and when is it a
“feature”?
And how do you know?
Uncertainty in Tes:ng (Current Research)
TestExecu:on
System
Under Test
Result Interpreta,on
Acceptable✓
Uncertainty in Tes:ng (Current Research)
TestExecu:on
System
Under Test
Result Interpreta,on
Unacceptable
Acceptable✓
✕
Uncertainty in Tes:ng (Current Research)
TestExecu:on
System
Under Test
Result Interpreta,on
Unacceptable
Acceptable
Acceptable
✓
✕
✕
Uncertainty in Tes:ng (Current Research)
TestExecu:on
System
Under Test
Result Interpreta,on
Unacceptable
Acceptable
Acceptable
✓
✕
✕
Acceptable misbehaviors can mask real faults!
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
TrainingData WEKA
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
WEKA
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
Result Interpreta,on
Acceptablep < 0.99
TestExecu:on
WEKA
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
Result Interpreta,on
Unacceptable
Acceptablep < 0.99
TestExecu:on
WEKA
p < 0.0027
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
Result Interpreta,on
Unacceptable
Acceptable
Inconclusive
p < 0.99
TestExecu:on
WEKA
p < 0.37
p < 0.0027
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
Conclusion
There is poten:ally much to be gained by relaxing the
tradi:onal absolu:st view of soKware engineering
And there are great research opportuni:es inapplying a probabilis:c viewpoint
– Norman Vincent Peale, The Power of Posi,ve Thinking
“Do not build up obstacles in your imagina:on.”