Download - Frontend Monitoring @ Grammarly
![Page 1: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/1.jpg)
Frontend Monitoring@Grammarly
![Page 2: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/2.jpg)
Grammarly Products
● Web editor - single page app
● Browser extensions for ○ Chrome, Safari, FireFox
● M$ Office Add-in
● Funnels
![Page 3: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/3.jpg)
![Page 4: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/4.jpg)
![Page 5: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/5.jpg)
![Page 6: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/6.jpg)
Grammarly Products :: Load
● 1M+ active users; 0.5M+ daily active
● 10M+ users planned next year
● 30 services on 300+ servers
● 130M+ requests/day
● 3.8M+ WebSocket connections/day
![Page 7: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/7.jpg)
What’s ur problem,bro?
![Page 8: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/8.jpg)
The problem
● Model != Reality
● 1B websites * X browsers
● Free users: Problem? => ⌘+W | Alt+F4
● Paid users: Problem? => Let’s torture support
![Page 9: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/9.jpg)
The problem :: CI / …
● Bugs
● Daily releases
● Performance testing
● A/B testing
![Page 10: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/10.jpg)
The Solution...
![Page 11: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/11.jpg)
The Solution :: Monitoring
● Monitoring != Tracking● Monitoring:
○ all data are volatile○ helps to assess quality in terms of
■ stability and ■ performance
○ fast problem detection and alerting○ troubleshooting○ different data sources incl. tracking events
![Page 12: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/12.jpg)
Grammarly FE Monitoring :: The Saga
● Manual testing● NewRelic● Errorception● Sentry● …● Profit (Custom Solution)
![Page 13: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/13.jpg)
FE Monitoring @ Grammarly
● Logging○ events with context (userId, UserAgent,
stacktraces, etc...)○ special cases only, no tracing ‘blah-blah’ logs
● TS Metrics○ everything else :)
● Alerting
![Page 14: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/14.jpg)
FE Monitoring Web browser
Elasticsearch x 4
Grafana Kibana
x 2Nginx
Access logs
LogstashStatsD
Graphite
x 2Sensu Checks
Sensu Server
OpsGenie
![Page 15: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/15.jpg)
Logging :: Backend Web browser
Elasticsearch x 4
Grafana Kibana
x 2Nginx
Access logs
LogstashStatsD
Graphite
x 2Sensu Checks
Sensu Server
OpsGenie
![Page 16: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/16.jpg)
TS Metrics :: Backend Web browser
Elasticsearch x 4
Grafana Kibana
x 2Nginx
Access logs
LogstashStatsD
Graphite
x 2Sensu Checks
Sensu Server
OpsGenie
![Page 17: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/17.jpg)
FE Monitoring
metrics codec
Logstash
metrics data~ 2600 RPS
~ 90 GiB / day
logs data~ 450 RPS
~ 50 GiB / day
Nginx access logs
StatsD
logs codec(+source maps)
ElasticSearch
tail *.log files
UDP HTTP
![Page 18: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/18.jpg)
FE Monitoring in numbers
● 38M logs/day
● up to 3K logs/sec @ busy hours
● ~100 Graphite metrics
● 6 servers + 2 shared w/ backend monitoring
![Page 19: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/19.jpg)
Logging :: JS Library
● legacy codebase from raven-js
● named loggers
● log levels (info, warn, error)
● default data in all events (aka MDC)
● scopes (lifetime, session, document)
![Page 20: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/20.jpg)
Logging :: JS Library
![Page 21: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/21.jpg)
![Page 22: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/22.jpg)
![Page 23: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/23.jpg)
![Page 24: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/24.jpg)
![Page 25: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/25.jpg)
![Page 26: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/26.jpg)
kibana screen
![Page 27: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/27.jpg)
![Page 28: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/28.jpg)
TS Metrics
● StatsD metrics: ○ counter (inc/dec)
○ timer: values for which StatsD calculates avg, min, max, percentile
○ set
![Page 29: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/29.jpg)
TS Metrics :: JS Library
Metric name: ui.performance.chrome.popup.loadCardinality is limited by Graphite storage (whisper)● product● version● browser● region (US | World)
![Page 30: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/30.jpg)
TS Metrics :: JS Library
![Page 31: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/31.jpg)
![Page 32: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/32.jpg)
TS Metrics :: UI
![Page 33: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/33.jpg)
![Page 34: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/34.jpg)
![Page 35: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/35.jpg)
![Page 36: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/36.jpg)
Case study● “Creeping” Versions● Active users● WebSocket errors● Stability● Performance● Page loads success/errors percentage● Bugs: …
![Page 37: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/37.jpg)
![Page 38: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/38.jpg)
![Page 39: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/39.jpg)
The Solution :: Adoption Problems● JS monitoring code bugs =>
○ wrong data○ self-DDoS
● FP alerts even on a correct data● Developers aren’t very passionate about writing logs
and metrics● Some education activity is needed to promote usage● turning monitoring into engineering practice
![Page 40: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/40.jpg)
The Solution :: Other Problems● Lack of data verification mechanism
● Graphite disk space issues
● High load as users base grows
● Monitoring infrastructure stability
![Page 41: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/41.jpg)
Near Future Plans● Graphite disk space scaling (Cassandra)● Client/server protocol optimization● Simple API for getting monitoring data for tests● Trends / New Events dashboards/facility● Simplified ES => Graphite metrics routing● Automatic code changes verification with A/B testing &
logs/metrics analysis
![Page 42: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/42.jpg)
Questions?
![Page 43: Frontend Monitoring @ Grammarly](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55ccf4ebbb61ebf7528b45be/html5/thumbnails/43.jpg)
Sergey Rudenko
Frontend engineer
fb://[email protected]@gmail.com
Aleksey Yashchenko
Backend engineer
fb://tuxslayer@[email protected]