ericsson falun 2015
TRANSCRIPT
Page 1
Using Varnish as Ski wax
Varnish Summit, 2015-10-22
Page 2
About me
➢ Lead developer of Ericsson Networked Event
➢ M.Sc. in Software Engineering from Chalmers
➢ Working at Ericsson since 2013
Page 3
About Falun 2015
➢ 3 different sports
– Cross country skiing
– Ski jumping
– Nordic combined➢ 714 competitors
➢ 12 days
➢ 21 competitions
➢ 281 600 on-site visitors
Page 4
Ericsson networked event
➢ A information broker specialized for events
– Distributing TV streams for competitions and music shows● Individual camera feeds● Main feeds
– Aggregating, transforming, analysing and distributing data as a RESTful API
● Live results and GPS-positioning of competitors● Historical results for competitors● Traffic and weather information● Schedule for both competitions and non-competition events● Etc.
Page 5
3D client Screen shots
Page 6
2D client Screen shots
Page 7
Platform overview
Data source
Data source
Data source
Data source
Clients
EricssonNetworkedEvent
Analysis Aggregation
Distribution
Transformation
Varnish
Page 8
Why Varnish?
➢ Very hard to estimate number of clients
– Varnish makes the load on the backend more predictable➢ Simplify scaling
– Scale Varnish horizontally without scaling the backend➢ Speeding up response times
– The application is complex with multiple data sources➢ Adding flexibility to the solution
– URL-rewriting / Routing
– Statistics
Page 9
Varnish in networked event
➢ Started with Varnish 3 in May 2014
➢ Migrated to Varnish 4 in autumn 2014
➢ Time based caching with grace in combination with bans
– Approx 1500 lines of VCL➢ Using Varnish High-Availability to distribute cache
➢ Got some tuning suggestions for RHEL from Varnish support
➢ TV-streams not handled through Varnish
Page 10
Varnish Panic
➢ February 18: Falun competitions starting
➢ February 19: Varnish 4.0.3 released
– Decided not to upgrade➢ February 26: Varnish panic due to failing backend fetch
– First response from Varnish support after 8 minutes
– Analysis done after 15 minutes
– Known bug (#1607), fixed in Varnish 4.0.3
Page 11
Varnish Panic
➢ 4.0.3 also had some improvements in memory handling
Page 12
Statistics summary
➢ 200 000 downloads of client apps
➢ 27 000 active daily users – from 167 countries
➢ 7000 TPS at peak
➢ 1.2 TB data handled
➢ All handled with two Varnish hosts running in HA
➢ Average hit rate: ~96 %
Page 13