predictive and adaptive failure mitigation to avert ... · predictive and adaptive failure...
TRANSCRIPT
Predictive and Adaptive Failure Mitigation to Avert Production Cloud VM Interruptions
OSDI ‘20
Sebastien Levy†, Randolph Yao†, Youjiang Wu†, Yingnong Dang†, Peng Huang^, Zheng Mu†, Pu Zhao*, Tarun Ramani†, Naga Govindaraju†, Xukun
Li†, Qingwei Lin*, Gil Lapid Shafriri†, Murali Chintalapati†
† Microsoft Azure, ^ Johns Hopkins University, * Microsoft Research
2
Azure global infrastructure
Azure regions
HeterogeneityAll nodes are different
Different characteristics: # cores, total memory …
Different hardware type / vendors
Different workload patterns
Different health history
3
Measuring Customer Experience
• We need to monitor VM availability: down time / up time
• But each VM interruption causes significant impact to customer• Disrupt user experience (ie gaming)• Applications take time to recover• Customer frustration would come in
case of repeated reboots• Two short interruptions are more
impactful than one longer one
4
Impact of Node Failure
VM VM
VM VM
VM
…
Disks CPUMemory
VM VM
VM VM
VM
…
Disks CPUMemoryFW/HW failure
• Node-level failure ➔ bad impact for every VM it contains
• We need to predict failures and take the appropriate mitigation actions
5
Traditional Operation Workflow
Offline & repair node
6
VM VM
VM VM
VM
…
Disks CPUMemory
VM VM
VM VM
VM
…
Disks CPUMemory
VM VM …
Healthy Node
2 Diagnosis
3 Migration
!
All VMs reboot
Long VM downtime
VM VM
VM VM
VM
…
Disks CPUMemory
1 Detection
?Node failure
Attempt 1: Mitigation Before Diagnostics
Offline & repair node
7
VM VM
VM VM
VM
…
Disks CPUMemory
VM VM
VM VM
VM
…
Disks CPUMemory
VM VM …
Healthy Node
1 Detection 3 Diagnosis
2 Migration
? !
All VMs reboot
Short VM downtime
Mitigation can be better in some cases
Node failure
VM VM
VM VM
VM
…
Disks CPUMemory
Attempt 2: Prediction of Node Failure
Offline & repair node
8
VM VM
VM VM
VM
…
Disks CPUMemory
VM VM
VM VM
VM
…
Disks CPUMemory
VM VM …
Healthy Node
1 Node predicted to fail 2 Diagnosis
3 Migration
?
VM VM
VM VM
VM
…
Disks CPUMemoryWait
Use prediction to speed up OS crash mitigation
All VMs reboot
Shorter VM downtime
Mitigation suboptimal for wrong prediction
Node failure
Attempt 3: Prediction + Static Mitigation
Offline & repair node
9
VM VM
Disks CPUMemory
VM VM …
Healthy Node
1 Node predicted to fail 3 Diagnosis
2
?
VM VM
Disks CPUMemoryBlock allocation
Capacity impact
Live migrate eligible VMsVM pause impact
Some VMs did not reboot
Other VMs reboot
Shorter VM downtime
Increased impact for false positives
Can we do better?
VM VM
VM VM
VM
…
Disks CPUMemory
Mitigation Tradeoff
❖ Can a mitigation be effective for all kind of scenarios?
❖ How to confirm a mitigation is effective?
❖ Even if a mitigation is effective, can we do better? Can it change in time?
10
Mitigation Tradeoff: No One-Size-Fits-All
How to mitigate predicted node failures: reasonable proposal
Capacity impactVM pause impact VM interruption impact
Block allocation
Live-migrate eligible VMs
Wait for 7 days
Force remaining VMs
migration
Diagnosis + Repair
If failure is not imminent, do we need to completely block
allocation?
How long should we wait for customers to intentionally move
their VMs?
If node has still not failed after 7 days, could it now be healthy?
11
How To Estimate Mitigation Efficacy?
• Heterogeneity: Multiple factors will impact the mitigation Efficacy• Live migration success rate depends on available capacity, hardware health,
workload on the node
• Allocation relies on a complex optimization logic and on stochastic customer demand
• Prediction false positives are unavoidable and can depend on unobservable signals
12
Key Insights
“In a heterogeneous and ever-changing cloud system, the effectiveness of a mitigation action is often probabilistic.”
“To select the (near-)optimal mitigation, each possible action needs to be compared at-scale with production workload.”
13
Solution
Continuous probabilistic online minimizationof customer impact
Different mitigation actions depending on the type of predicted failure
Online testing of the different options
Adapt to ever-changing cloud behavior
Account for node heterogeneity of Azure’s fleet
Focus on impacting failures
14
Narya’s Overview
Prediction Rule
ML failure prediction Customer impact
Feedback loop
Diagnostics
Offline Monitoring
…
CostLabels
Prediction Decision Mitigation Impact Assessment15
1
3
2
1
4
4
Choose action
Main Challenges
Define Define customer impact as a generic metric
Test Safely test action in production
BalanceBalance exploring actions and exploiting the best so far
Adapt Adapt our decision to system changes
Fast Fast and scalable mitigation decision
16
Measuring Interruptions
• Interruption Types:
• VM reboot, IO pause, VM temporarily freeze / blackout
• Service Availability
• Time duration based
• Short VM downtime DOES NOT imply short customer service downtime.
• Each VM service downtime requires customer service to rehydrate their state
• Interruption Count
• Event count based
• Each VM interrupt impacts customer service once
• More realistically reflects the customer pain
17
Defining Customer Impact
• VM interruptions are the main negative impact to customer
• We define Annual interruption rate (AIR)
𝐴𝐼𝑅 =# 𝑖𝑛𝑡𝑒𝑟𝑟𝑢𝑝𝑡𝑖𝑜𝑛𝑠 𝑖𝑛 𝑇
𝑇𝑜𝑡𝑎𝑙 𝑢𝑝 𝑡𝑖𝑚𝑒 𝑖𝑛 𝑇× 365 𝑑𝑎𝑦𝑠 × 100
Cost = # VM interruptions per node
When scoping it to a contribution per node and removing the constants, we get:
18
Narya’s Prediction
Expert Rules ML Prediction
19
Motivation To Use ML Prediction
• Expert rule • Setting a threshold on a single predictive signal. Ex: Disk low spare space
• Limitations of expert rule• High variance of prediction horizon - actual failure time also depends on factors
• Difficult precision-recall tradeoff – strict rules result in low recall, loose rules yield low precision
• What if we look at all signals before the actual failure and use those signals to predict the failure?
20
ML Model Highlights
• Comprehensive list of signals that cover OS layer, driver layer and device layer observation.
• Predict server level failures that has customer impact
• Non-handcrafted features• Spatial attention
• Temporal convolution
21
Longer Lead Time
Higher Precision
Higher Recall
Diversity of Mitigation Actions
Allocation change VM migration Node reboot
Allow allocation
Avoid allocation*
Block allocation
Live migrate VMs*(unallocatable node only)
Service heal VMs(unallocatable node only)
Soft kernel reboot*(unallocatable node only)
Hard reboot
Offline + repair(empty node only)
*
No Impact
Pause Impact
Soft capacity impact
Hard capacity impact
VM reboot impact
Non-deterministic action22
Narya’s Mitigation Actions
• Use composite actions (sequence of actions)• Avoid illogical scenarios: migration without blocking allocation, repeated
reboots …
• Add safety constraint to minimize the cost of exploration
• Easy override priority if several rules overlap
• Simpler learningIf OS crashes
If empty
Example:
Composite actions’ duration is typically in the order of days 23
Narya’sDecision Engine
• Mapping each predicted bad nodes to the best composite mitigation action to minimize customer pain
• Two algorithms:• AB testing:
explore all different possibilities, observe the customer pain metric(s) then use the best action if it exists
• Multi-Armed Bandit (Thompson Sampling): learn the right tradeoff between exploring new actions and exploiting the estimated best one based on a single customer pain metric
24
AB Testing
1. Sample action with preconfigured probabilities
2. Observe customer impact within an observation window
3. Hypothesis test between cost of all actions
4. If statistical significance, use the optimal action
Action A
Compare statistical impact
Action B25
Adapting AB Testing To Our Scenario
• Cost attribution: VM reboots during the observation window• knowing which interruptions is caused by the action is not possible
• Stickiness: same node always use the same action• otherwise we would violate the iid assumption
• Decision vs action: we observe the consequence of every decision• even if it is skipped, delayed or overridden
26
Bandit Motivation
• AB testing limitations
• Does not leverage early observation before statistical significance
• Cannot adapt to change after statistical significance
• Multi-armed bandit: dynamically learn the probability based on observations
✓Leverage early observation in exploration
✓ Adapt to change in exploitation
27
Bandit Framework
Reward = - Customer impact (Cost)
Machines = Available composite actions
Pull / Game = New node mitigation request
Each prediction rule is a different bandit experiment
28
Bandit Adaptation To Our Scenario
Accommodate temporal change: exponential decay
Delayed rewards
Bandit stickiness Safeguards
29
Ensuring Safe Exploration
• Only allow relevant composite actions
• Override logic if other more severe issues are detected
• Fallback to AB testing if there are not enough observation data
• Enforce minimum and maximum probability for each action
30
31
Mitigation Engine
ML Prediction
Pub / Sub Real-time
Node Monitoring
Agent
Action Orchestrator
Policy Generator
Req
ues
t H
and
ler
Model Serving Platform Learner
VM Impact
Mitigation decision
Narya’s System Architecture1
3
2
Scalable
Fast
Adaptive
Overall Savings: 26%
• Compare the use of AB testing/Bandit to previous static strategy
• Estimated AIR savings: [Observed AIR] – [Control group AIR mapped to whole fleet]
0
10
20
30
40
50
60
% Im
pro
vem
ent
Experiments
Bandit improvement over AB testing
32
Prediction Performance
• Precision – 79.5%
• Recall – 50.7%
• ML prediction:Time to failure – 48 hours on average
33
Compare AB Testing and Bandits
• Counterfactual estimation
18.4
7.7
22.3
16.6
3.8
7.1
15.3
0
5
10
15
20
25
2 Ierr E500 E11 Tardigrade E7 30 Threshold 1 Ierr 63023 Orange ML Prediction
Improvement (%)34
Improvement (%)
Case Study: I/O Timeouts
• A/B testing then Bandit experiment on I/O timeouts prediction rule
• Unexpected system changes switched probability from using NoOp to using UA-LM-RH automatically
• Although change is not understood, bandit can automatically adjust
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2/1
8/2
02
0
2/2
0/2
02
0
2/2
2/2
02
0
2/2
4/2
02
0
2/2
6/2
02
0
2/2
8/2
02
0
3/1
/20
20
3/3
/20
20
3/5
/20
20
3/7
/20
20
3/9
/20
20
3/1
1/2
02
0
3/1
3/2
02
0
3/1
5/2
02
0
3/1
7/2
02
0
3/1
9/2
02
0
3/2
0/2
02
0
3/2
2/2
02
0
3/2
4/2
02
0
3/2
6/2
02
0
3/2
8/2
02
0
3/3
0/2
02
0
4/1
/20
20
4/3
/20
20
4/5
/20
20
4/7
/20
20
Bandit probability
UaProb NoOpProb
35
UA-LM-RH: Unallocatable + Live Migration + Reset Node Health
Operation Learnings
• Many factors influence efficiency: need for probabilistic approaches
• Decisions may go wrong: closely monitor all components behavior and use interpretable models
• Data quality is critical: watch for telemetry schema changes
• Customer impact is complex: human in the loop helps prevent from new types of impact
36
Summary / Takeaway Message
• Both failure prediction and proactive mitigation is critical
• No one-size-fits-all mitigation strategy, adapting different mitigation strategies in an online fashion
• Narya uses AB testing and multi-armed bandit to proactively and adaptively mitigate of predicted bad nodes
• Narya achieved 26% improvement over previous static strategy
37
Thank you!
• Acknowledgement • Haryadi Gunawi, our shepherd, and the anonymous reviewers
• All our collaborators within Azure
• Contact us• [email protected]
38