one network engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on...
TRANSCRIPT
![Page 1: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/1.jpg)
ONE
James Paussanetwork infrastructure engineer
One Network Engineer
![Page 2: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/2.jpg)
What Do We Do?
• A single engineer runs the network at a time
• Network monitoring and alarming
• Responsible for the entire production network
![Page 3: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/3.jpg)
Facebook Scale
968 million daily active users on average
844 million mobile daily active users on average
1.31 billion mobile monthly active users
1.49 billion monthly active users
![Page 4: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/4.jpg)
Machine to machine
Machine to user
Facebook Scale
![Page 5: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/5.jpg)
Automation
![Page 6: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/6.jpg)
What We Don’t Do
• On site work (cleaning fibres, LCs, etc)
• Working with remote hands
• Device deployment
![Page 7: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/7.jpg)
Myths
• Automation fixes everything
• We’ve fixed everything
• Doesn’t apply
![Page 8: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/8.jpg)
Link Imbalance
![Page 9: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/9.jpg)
Link Imbalance
![Page 10: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/10.jpg)
Link Imbalance100
50
0
Interface Utilization
![Page 11: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/11.jpg)
Link Imbalance
![Page 12: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/12.jpg)
Link Imbalance100
50
0
Interface Utilization
![Page 13: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/13.jpg)
Link Imbalance100
50
0
Interface Utilization
![Page 14: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/14.jpg)
Link Imbalance
srsly wtf.
![Page 15: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/15.jpg)
Link Imbalance
• Intercluster links only
• >80%+ Util
• Migratory
![Page 16: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/16.jpg)
What Now?
• Detection
• Mitigation
![Page 17: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/17.jpg)
Imbalance Detection
Aggregated Interfaces
Member Statistics
Compare Utilisation
Check Fails Notify Oncall
![Page 18: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/18.jpg)
Imbalance Mitigation
SIP DIP SPORT DPORT Hash Key
![Page 19: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/19.jpg)
Imbalance Mitigation
Roll Hash
![Page 20: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/20.jpg)
Imbalance Mitigation
![Page 21: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/21.jpg)
Link Imbalance
Roll HashDetect ResolvedFBAR
![Page 22: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/22.jpg)
![Page 23: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/23.jpg)
Link Imbalance
ZOMG!! WTF!! NO, NO, NO! &(#$&*(
![Page 24: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/24.jpg)
Link Imbalance
RX Window
During
![Page 25: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/25.jpg)
Link Imbalance
Open TCP connections
During After
![Page 26: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/26.jpg)
Cache DB
It Isn’t (always) The Network
![Page 27: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/27.jpg)
Cache DB
![Page 28: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/28.jpg)
Cache DB
![Page 29: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/29.jpg)
Cache DB
![Page 30: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/30.jpg)
Link Imbalance - Lessons Learned
• Resolved issues
• Root Cause
• Software helps
• Service owner identified
• Resolution time
• Small loss, significant impact
![Page 31: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/31.jpg)
3 Months In A Leaky Boat
![Page 32: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/32.jpg)
!Capacity
Health
Errors
Memory Issues
![Page 33: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/33.jpg)
DIP Loss Latency
1.1.1.1 0.1 10
2.2.2.2 0 10
DIP Loss Latency
1.1.1.1 0.1 10
Servers ServersServers
Servers ServersServers
Detection
![Page 34: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/34.jpg)
Loss Effects on Throughput1000
400
00.00000% 0.00001% 0.00010% 0.00100% 0.01000% 0.10000% 1.00000%
Packet Loss %
Thro
ugh
pu
t (m
bp
s) 800
600
200
RTT0
X20025
XXXX
XX
![Page 35: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/35.jpg)
XX
XX X X0
0.00000% 0.00001% 0.00010% 0.00100% 0.01000% 0.10000% 1.00000%
Packet Loss %
Thro
ugh
pu
t (m
bp
s)
350
50
100
150
250
300
200
Different algos?
X
RenoCubicVegasIllinois
![Page 36: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/36.jpg)
Recovery time
0
Thro
ugh
pu
t (m
bp
s)
350
50
100
150
250
300
200
Time (sec)0 120110100908070605040302010
1% P
acke
t Los
s RenoCubicVegasIllinois
![Page 37: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/37.jpg)
So, wait, how does this apply to me?
![Page 38: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/38.jpg)
Alarms
0
70k
70k!
![Page 39: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/39.jpg)
!Capacity
Health
Errors
Interface Issues
!!!!!!!!!!!…………………………………………………..
![Page 40: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/40.jpg)
Alarms Now
0
70
<100 -99.99%
![Page 41: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/41.jpg)
Automation - One Month
3.37b0.99%
750k99.6%
![Page 42: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/42.jpg)
Why?
Sleep
![Page 43: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/43.jpg)
Why?
750,000 * 2 = 1,500,0001,500,000 / 60 = 25,000
25,000 / 160150+
![Page 44: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/44.jpg)
Lessons Learned & Take AwaysUse what you’ve got
Prototype early, iterate often
Duct tape keeps things running
Spend the time to root cause issues
The sooner the robots take over the better
![Page 45: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/45.jpg)
Why?
![Page 46: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/46.jpg)
![Page 47: One Network Engineer - ausnog.net · 2015. 8. 25. · 844 million mobile daily active users on average 1.31 billion mobile monthly active users 1.49 billion monthly ... • Working](https://reader034.vdocuments.mx/reader034/viewer/2022052023/60384ecc580efa1470291197/html5/thumbnails/47.jpg)
?