towards verified artificial intelligencesseshia/219c/spr...formally specify the end‐to‐end...
TRANSCRIPT
![Page 1: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/1.jpg)
Towards Verified Artificial Intelligence
Sanjit A. SeshiaUC Berkeley
EECS 219CApril 24, 2019
![Page 2: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/2.jpg)
Growing Use of Machine Learning/AI in Cyber‐Physical Systems
S. A. Seshia 2
Many Safety‐Critical Systems
![Page 3: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/3.jpg)
S. A. Seshia 3[NTSB]
![Page 4: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/4.jpg)
How do we ensure that AI/ML‐based systems are Dependable?
S. A. Seshia 4
Artificial Intelligence (AI)
Computational Systems that attempt to mimic aspects of human intelligence, including especially the ability to learn from experience.
![Page 5: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/5.jpg)
Challenges for Verified AI
S. A. Seshia 5
System SEnvironment ESpecification
YES [+ proof]Does S || E satisfy ?
NO [+ counterexample]
S. A. Seshia, D. Sadigh, S. S. Sastry. Towards Verified Artificial Intelligence. July 2016. https://arxiv.org/abs/1606.08514.
![Page 6: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/6.jpg)
Environment Modeling Challenge – Uncertainty and Unknowns
S. A. Seshia 6
Self‐Driving Vehicles: Interact with Humans in Complex Environments;Significant use of machine learning!
Known Unknowns andUnknown Unknowns!!
Cannot represent all possible environment scenarios
![Page 7: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/7.jpg)
Challenges for Verified AI
S. A. Seshia 7
System SEnvironment ESpecification
YES [+ proof]Does S || E satisfy ?
NO [+ counterexample]
S. A. Seshia, D. Sadigh, S. S. Sastry. Towards Verified Artificial Intelligence. July 2016. https://arxiv.org/abs/1606.08514.
![Page 8: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/8.jpg)
Modeling Learning Systems with High‐Dimensional Input & State Space
S. A. Seshia 8
Histogram of(label, confidence)Stream of images
Input Space: ~106 dimensions for single time pointSystem Parameters: >1M, continuous+discrete
Need New Methods for Abstraction and Modular Reasoning!
![Page 9: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/9.jpg)
Challenges for Verified AI
S. A. Seshia 9
System SEnvironment ESpecification
YES [+ proof]Does S || E satisfy ?
NO [+ counterexample]
S. A. Seshia, D. Sadigh, S. S. Sastry. Towards Verified Artificial Intelligence. July 2016. https://arxiv.org/abs/1606.08514.
![Page 10: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/10.jpg)
What’s the Specification for Perception Tasks?
S. A. Seshia 10
Convolutional Neural Network trained to recognize cars
How do you formally specify “a car”?
![Page 11: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/11.jpg)
Challenges for Verified AI
S. A. Seshia 11
System SEnvironment ESpecification
YES [+ proof]Does S || E satisfy ?
NO [+ counterexample]
S. A. Seshia, D. Sadigh, S. S. Sastry. Towards Verified Artificial Intelligence. July 2016. https://arxiv.org/abs/1606.08514.
Design Correct‐by‐Construction instead? How?
Counterexamples, Inputs, etc. from High‐Dimensional Signal Spaces
![Page 12: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/12.jpg)
Need Design Principles for Verified AIChallenges
1. Environment (incl. Human) Modeling
2. Formal Specification
3. Learning Systems Representation
4. Scalable Training, Testing, Verification
5. Design for Correctness
Principles
?
S. A. Seshia 12
S. A. Seshia, D. Sadigh, S. S. Sastry. Towards Verified Artificial Intelligence. July 2016. https://arxiv.org/abs/1606.08514.
![Page 13: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/13.jpg)
Outline
• Challenges for Verified AI
• Design of Closed‐Loop Cyber‐Physical Systems with Machine Learning Components– Specification, Verification, Synthesis– Autonomous Vehicles– Deep Learning
• Principles for Verified AI – Summary of Ideas– Future Directions
S. A. Seshia 13
![Page 14: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/14.jpg)
Challenge: Formal Specification
S. A. Seshia 14
Principle: Start at System Level (i.e. Specify Semantic Behavior of the
Overall System)
![Page 15: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/15.jpg)
Falsification of Cyber‐Physical Systems with Machine Learning Components
S. A. Seshia 15
T. Dreossi, A. Donze, and S. A. Seshia. Compositional Falsification of Cyber-Physical Systems with Machine Learning Components, In NASA Formal Methods Symposium, May 2017.
![Page 16: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/16.jpg)
Problem: Verify Cyber‐Physical System that uses ML‐based Perception (CPSML system)
16
• Initial Focus:• Falsification: finding scenarios that violate safety properties• Test (Data) Generation: generate “interesting” data for
training / testing improve accuracy • Deep Neural Networks, given the increasing interest and use
in the automotive context.
S. A. Seshia
Controller Plant
Environment
Learning‐Based Perception
Sensor Input
x y
sp
usp
se
f
![Page 17: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/17.jpg)
Automatic Emergency Braking System (AEBS)
17
AEBS Controller Plant
Environment
Deep Learning‐Based Object Detection
• Goal: Brake when an obstacle is near, to maintain a minimum safety distance• Initial: Controller, Plant, Env models in Matlab/Simulink• More recent: other driving simulators
• Perception: Object detection/classification system based on deep neural networks • Inception‐v3, AlexNet, … trained on ImageNet• more recent: squeezeDet, Yolo, … trained on KITTI
S. A. Seshia
![Page 18: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/18.jpg)
S. A. Seshia 18
Our Approach: Use a System‐Level Specification
“Verify the Deep Neural Network Object Detector”
“Verify the System containing the Deep Neural Network”
Formally Specify the End‐to‐End Behavior of the System
Controller Plant
Environment
Learning‐Based Perception
Temporal Logic: G (dist(ego vehicle, env object) > )
Property does not mention inputs/outputs of the neural network(sp,se,u)
x y
sp
usp
se
![Page 19: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/19.jpg)
Temporal Logic Requirements
Signal Temporal Logic (STL)Predicates over continuous signals, Propositional Formulas (∧,∨, of the predicates), Temporal Operators ( , , , ), real-time interval .
is true at all future moments in .is true in some future moment in .is true until becomes true in .
, [ dist(vehicle, obstacle) > ]
[Maler & Nickovic, ‘04]
Safety (invariance): Vehicle maintains specified min distance from obstacles.
19S. A. Seshia
![Page 20: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/20.jpg)
From Logical Formulas to Objective Functions
• STL formula has both– Boolean semantics: true/false– Quantitative semantics: value in
• Example: , (dist(vehicle, obstacle) > )
inf[0, ] [ dist(vehicle, obstacle) - ]
20S. A. Seshia
![Page 21: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/21.jpg)
Specification Spectrum
• System Level (“Semantic”)– Typical properties of reactive systems (safety, liveness, etc.)
• Component Level – Robustness– Input‐Output Relations– Monotonicity– Fairness– Coverage– …
S. A. Seshia 21
[S. A. Seshia, et al., “Formal Specification for Deep Neural Networks”, TR May 2018; ATVA’18]
![Page 22: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/22.jpg)
Robustness Analysis
S. A. Seshia 22
![Page 23: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/23.jpg)
Robustness as a Property
S. A. Seshia 23
[Dreossi, Ghosh, Sangiovanni‐Vincentelli, Seshia, VNN 2019]
Optimization Versions
![Page 24: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/24.jpg)
Summary of Robustness Properties
S. A. Seshia 24
[Dreossi, Ghosh, Sangiovanni‐Vincentelli, Seshia, VNN 2019]
![Page 25: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/25.jpg)
Specification Spectrum
• System Level (“Semantic”)– Typical properties of reactive systems (safety, liveness, etc.)
• Component Level – Robustness– Input‐Output Relations– Monotonicity– Fairness– Coverage– …
S. A. Seshia 25
[S. A. Seshia, et al., “Formal Specification for Deep Neural Networks”, TR May 2018; ATVA’18]
Need a notion of Semantic Robustness Robustness in the Semantic Feature Space [Dreossi, Jha, Seshia CAV 2018]
![Page 26: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/26.jpg)
Challenge: Scalability of Verification Principle: Compositional Simulation‐Based Verification (Falsification)
S. A. Seshia 26
![Page 27: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/27.jpg)
CPSML model
S. A. Seshia 27
Controller Plant
Environment
Learning‐Based Perception
x y
sp
usp
se
Traditional CPS model
Controller Plant
Environment
sp
usp
se
Challenge: se << x
![Page 28: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/28.jpg)
Three Key Ideas
1. Reduce CPSML falsification problem to combination of CPS falsification and ML analysis
2. Simulation‐based temporal logic falsification for CPS model
3. ML analysis via systematic exploration of “semantic feature space”
S. A. Seshia 28[Dreossi et al., NFM’17; Dreossi et al., CAV’18]
![Page 29: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/29.jpg)
Simulation‐Based Falsification of Signal Temporal Logic for CPS
S. A. Seshia 29
• STL has quantitative semantics– Logical formula Cost Function – Quantifies “how much” a trace satisfies a property
• Advantage: Finding a bug (property violation) minimizing and checking if value < 0.– View of “verification as optimization” underlies simulation‐based falsification tools
– Used by some production groups in automotive industry
• Can also apply to MTL and other TL variants
![Page 30: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/30.jpg)
Need Compositional Falsification of CPSML
S. A. Seshia 30
• However: no formal spec. for neural network!• Compositional Verification without Compositional Specification?!!
• Challenge: Very High Dimensionality of Input Space!• Standard solution: Use Compositional (Modular)Verification
Controller Plant
Environment
Learning‐Based Perception
x y
sp
usp
se
(see [Seshia, UCB/EECS TR 2017])
![Page 31: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/31.jpg)
Compositional Approach: Combine Temporal Logic CPS Falsifier with ML Analyzer
31S. A. Seshia
CPSML model MProperty
[Dreossi, Donze, Seshia, NFM 2017]
AbstractML component away from M
Overapproximate MUnderapproximate M
Invoke CPS Falsifier(multiple times)
Region of Uncertainty ROU(se,sp,u)
Component(ML) Analysis
Component‐level errors(misclassifications)
Refine
Project to ML Feature Space
FS(x)
where ML decision matters
Counterexample(s)Full CPSML Simulation
![Page 32: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/32.jpg)
Identifying Region of Uncertainty (ROU) for Automatic Emergency Braking System
32S. A. Seshia
ML always correct ML always wrong Potentially unsafe region depending on ML
component (yellow)
Green environments where the property is satisfied
Underapproximate M Overapproximate M
![Page 33: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/33.jpg)
Machine Learning Analyzer
33
Systematically Explore ROU in the Image (Sensor) Space
Semantic Feature space
brightness car z-pos
Abstraction map
brightnesscar z-pos
car x-pos
Abstract space A
S. A. Seshiax
Abstract space A
Neural network ∈ ,
✓
✓ ✕
✕
✕
✓
✓✓
✓
✕
Systematic Sampling (low‐discrepancy sampling)
![Page 34: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/34.jpg)
Sample Result
34
Inception‐v3Neural Network
(pre‐trained on ImageNet using TensorFlow)
Misclassifications
This misclassification may not be of concern
S. A. Seshia
![Page 35: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/35.jpg)
Sample Result
35
Inception‐v3Neural Network
(pre‐trained on ImageNet using TensorFlow)
Misclassifications
Corner caseImage
But this one is a real hazard!
S. A. Seshia
![Page 36: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/36.jpg)
Image Streams
S. A. Seshia 36
Superimposition of tests on backgroundBlind spots
Results on squeezeDet NN and KITTI dataset for autonomous driving
[Dreossi, Ghosh, et al., ICML 2017 workshop]
![Page 37: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/37.jpg)
Summary
• Write System‐Level Specifications• Use Compositional Approach
– Falsification on the CPSML Model– Extends to Verification in some cases
• Approach Generates Counterexamples
How do we use these counterexamples?
Semantic Adversarial Machine (Deep) Learning [Dreossi, Jha, Seshia, CAV 2018]
S. A. Seshia 37
![Page 38: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/38.jpg)
Challenge: How to (Re)Design Learning Components
Principle: UseOracle‐Guided Inductive Synthesis
S. A. Seshia 38
![Page 39: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/39.jpg)
Correct‐by‐Construction Design with Formal (Oracle‐Guided) Inductive Synthesis/Learning
Inductive Synthesis: Learning from Examples (ML)Formal Inductive Synthesis: Learn from Examples while satisfying a Formal Specification
S. A. Seshia 39
[Jha & Seshia, “A Theory of Formal Synthesis via Inductive Learning”, 2015,Acta Informatica 2017.]
Key Idea: Oracle‐Guided LearningCombine Learner with Oracle (e.g., Verifier) that answers Learner’s Queries
LEARNER ORACLE
query
response
![Page 40: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/40.jpg)
Counterexample‐Guided Training of Deep Neural Networks• Instance of Oracle‐Guided Inductive Synthesis• Oracle is Verifier (CPSML Falsifier) used to find counterexample inputs to DNN
• Substantially increase accuracy with relatively few additional examples
S. A. Seshia 40
DEEP NEURAL NETWORK
FALSIFIER (CPS + ML)
Learned Classifier
“Counterexample‐Guided Data Augmentation”, T. Dreossi, S. Ghosh, X. Yue, K. Keutzer, A. Sangiovanni‐Vincentelli, S. A. Seshia, IJCAI 2018.
![Page 41: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/41.jpg)
Counterexample‐Guided Data Augmentation
S. A. Seshia 41
Id Car color Background Orientation
1 Red Countryside Front
2 Orange Forest Back
3 White Forest Front
4 Green Forest Back
Train Aug.
“Counterexample‐Guided Data Augmentation”, T. Dreossi, S. Ghosh, X. Yue, K. Keutzer, A. Sangiovanni‐Vincentelli, S. A. Seshia, IJCAI 2018.
![Page 42: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/42.jpg)
Experimental Results
S. A. Seshia 42
“Counterexample‐Guided Data Augmentation”, T. Dreossi, S. Ghosh, X. Yue, K. Keutzer, A. Sangiovanni‐Vincentelli, S. A. Seshia, IJCAI 2018.
Model Precision RecallOriginal .61 .75Standard augmentation .69 .80Uniform random .76 .87Constrain .75 .86Low-discrepancy .79 .87Cross-entropy .78 .78
Train - 1.5k
Counterexamples
Comparison of Augmentation Methods
CTest - 0.75k Aug - 0.75k
Model Precision RecallOriginal .61 .75Standard augmentation .69 .80
Model Precision RecallOriginal .61 .75
Counterexample-guided
augmentation
Test - 0.75k
![Page 43: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/43.jpg)
Experimental Results
S. A. Seshia 43
Model TestOriginal 0.98Model Test CTest 1Original 0.98 0.69Model Test CTest 1Original 0.98 0.69Aug1 0.99 0.82
Model Test CTest 1 CTest 2Original 0.98 0.69Aug1 0.99 0.82 0.59
Model Test CTest 1 CTest 2 CTest 3Original 0.98 0.69Aug1 0.99 0.82 0.59Aug2 0.99 0.86 0.82 0.53
Model Test CTest 1 CTest 2 CTest 3Original 0.98 0.69Aug1 0.99 0.82 0.59Aug2 0.99 0.86 0.82 0.53Aug3 0.99 0.84 0.80 0.88
Time (h)0.02Time (h)0.02~6
Time (h)0.02~6~14
Time (hrs)0.02~6~14~26
Precision – Low-discrepancy sampling
Train - 1.5k
Counterexamples
CTest - 0.75k Aug - 0.75k
Much harder to find counterexamples after retraining!!!
Test - 0.75k
![Page 44: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/44.jpg)
Principles: Data‐Driven, Introspective, Stochastic Modeling
S. A. Seshia 44
Challenge: Environment Modeling capturing assumptions
Environment
Sensor Input
x se
![Page 45: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/45.jpg)
Generating Meaningful (Sensor) Data
S. A. Seshia 45
• Large and unstructured input space
• We want scenes that make physical sense and are interesting and useful for training/testing or design
• How can we guide data generation towards such scenes?
Car Model Car Location Car Orientation
Number of Cars Reference Scene Background
Car Color Weather Time of Day
Images created with GTA‐V
Our approach: Scene Improvisation
[D. Fremont et al., “Scenic: Language‐Based Scene Generation”, 2018.]
![Page 46: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/46.jpg)
SCENIC: Scenario Description Language
S. A. Seshia 46
• Scenic is a probabilistic programming language defining distributions over scenes
• Example scenario: a badly-parked car
[D. Fremont et al., “Scenic: Language‐Based Scene Generation”, 2018.]
![Page 47: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/47.jpg)
SCENIC: a Scenario Description Language
• Defines a distribution over scenes
• Readable, concise syntax for common geometric relationships
• Declarative hard and softconstraints
• Scenarios on a spectrum from very broad classes of scenes to small variations on a single scene
47
![Page 48: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/48.jpg)
SCENIC: Scenario Description Language
S. A. Seshia 48
Scenic makes it possible to specify broad scenarios with complex structure, then generate many concrete instances from them automatically:
Platoons Bumper-to-Bumper Traffic
![Page 49: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/49.jpg)
Use Case: Retraining with Hard Cases
S. A. Seshia 49
e.g. for car detection, one car partially occluding another:
![Page 50: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/50.jpg)
Use Case: Retraining with Hard Cases
S. A. Seshia 50
e.g. for car detection, one car partially occluding another:
Improves accuracy on hard cases without compromising accuracy on original training set
![Page 51: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/51.jpg)
Use Case: Generalizing a Known Failure
S. A. Seshia 51
![Page 52: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/52.jpg)
Use Case: Generalizing a Known Failure
S. A. Seshia 52
Scenic makes it easy to generalize along different dimensions:
Add noise Change car model Change global position
![Page 53: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/53.jpg)
Outline
• Sample Challenges for Verified AI
• Design of Closed‐Loop Cyber‐Physical Systems with Machine Learning Components– Specification, Verification, Synthesis– Autonomous Vehicles– Deep Learning
• Principles for Verified AI – Summary of Ideas– Future Directions
S. A. Seshia 53
![Page 54: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/54.jpg)
VERIFAI: A Toolkit for the Design and Analysis of AI‐Based Systems [Dreossi, Fremont, Ghosh, et al., CAV 2019]
https://github.com/BerkeleyLearnVerify/VerifAI
![Page 55: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/55.jpg)
Environment Modeling: the SCENIC Language
Fremont et al., Scenic: A Language for Scenario Specification and Scene Generation, PLDI 2019 (to appear).
![Page 56: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/56.jpg)
![Page 57: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/57.jpg)
![Page 58: Towards Verified Artificial Intelligencesseshia/219c/spr...Formally Specify the End‐to‐End Behaviorof the System Controller Plant Environment Learning‐Based Perception Temporal](https://reader034.vdocuments.mx/reader034/viewer/2022052100/60395b650e46011a773fd896/html5/thumbnails/58.jpg)
Towards Verified Artificial Intelligence
S. A. Seshia 58
S. A. Seshia, D. Sadigh, S. S. Sastry. Towards Verified Artificial Intelligence. July 2016. https://arxiv.org/abs/1606.08514.
Challenges
1. Environment (incl. Human) Modeling
2. Specification
3. Learning Systems Complexity
4. Scalable Training, Testing, Verification
5. Design for Correctness
Principles
Data‐Driven, Introspective, Stochastic ModelingStart with System‐Level Spec.; Derive Component Specs. Abstraction, Semantic Feature Analysis, ExplanationsCompositional Analysis and Algorithmic ImprovisationFormal, Oracle‐Guided Inductive Synthesis; Run‐time Assurance