faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans
Post on 18-Dec-2015
222 views
TRANSCRIPT
![Page 1: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/1.jpg)
Faz.net technical challenges 2000 - 2009
![Page 2: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/2.jpg)
what i won’t show
• business numbers, €• business plans
![Page 3: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/3.jpg)
what to do (from the point of view of lowly programmer)
• invent a business plan• define, plan, and implement desired functionality• build a robust, secure, scalable system• keep that system up 100% of the time• allow changes to support new business initiatives
![Page 4: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/4.jpg)
a brief history
• launched 8 Jan 2001• “stable” by Spring of 2001• unstable on 11 Sep 2001• better able to handle (many classes of) huge load increases by
Spring 2002
![Page 5: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/5.jpg)
![Page 6: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/6.jpg)
how big is faz.net?
• much smaller than google • smaller than spiegel
![Page 7: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/7.jpg)
![Page 8: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/8.jpg)
how big is faz.net?
• much smaller than google • smaller than spiegel• comparable to other newspapers
![Page 9: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/9.jpg)
![Page 10: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/10.jpg)
![Page 11: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/11.jpg)
![Page 12: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/12.jpg)
![Page 13: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/13.jpg)
a sketch of our system layout (a couple years old)
• load balancer• web servers• DB and file servers• other application servers• client machines, e.g. newsroom• external partners, e.g. freemail provider
![Page 14: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/14.jpg)
![Page 15: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/15.jpg)
![Page 16: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/16.jpg)
an ancient attempt to show how changing an article affects site
• editor publishes a new article (or a new version of an existing article)
• a DB trigger fires• cached DB result sets get updated• cached HTML gets updated or deleted• (we do things differently now)
![Page 17: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/17.jpg)
![Page 18: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/18.jpg)
![Page 19: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/19.jpg)
how to measure what is happening / whether things are ok
• measure throughput at / between various points
![Page 20: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/20.jpg)
(partial) network traffic over time
• MRTG (http://oss.oetiker.ch/mrtg/)
• Standard, free (Gnu)
• Useful at a glance info
![Page 21: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/21.jpg)
how to measure what is happening / whether things are ok
• measure throughput at / between various points• measure, e.g. cpu load on web servers (NB: load is pretty low, max
would be 12*100)
![Page 22: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/22.jpg)
![Page 23: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/23.jpg)
![Page 24: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/24.jpg)
![Page 25: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/25.jpg)
long term trend: one year
![Page 26: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/26.jpg)
![Page 27: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/27.jpg)
Some errors can be seen by any user with a browser who runs into them
• Server error• Wrong contents• Broken HTML, images, or …
![Page 28: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/28.jpg)
some page ailments complain helpfully
![Page 29: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/29.jpg)
![Page 30: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/30.jpg)
![Page 31: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/31.jpg)
digested log files can be helpful
• web sites tend to have many log files, several kinds of log files; and some kinds can be *huge*
• they tend to be straight ascii files in some format you might have little control over
• various kinds of statistics might sometimes interest you• in particular error statistics
![Page 32: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/32.jpg)
ad hoc (logfiles | awk ... excel
![Page 33: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/33.jpg)
ad hoc: mrtg + paint o.ä.
![Page 34: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/34.jpg)
ad hoc
![Page 35: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/35.jpg)
ad hoc: DB query -> Excel
![Page 36: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/36.jpg)
external, hired monitoring service
• hire someone outside your site to watch certain pages on your site (load them periodically) and keep statistics about timing, sizes, errors
• ideally get regular reports showing everything is groovy• support “drilling” to get more details when necessary
![Page 37: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/37.jpg)
![Page 38: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/38.jpg)
using logarithms to graph events with highly varying scales
![Page 39: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/39.jpg)
some 50x errors showed up on utility servers one day
![Page 40: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/40.jpg)
![Page 41: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/41.jpg)
a primitive top 10 most popular pictures (PIs per hour)
![Page 42: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/42.jpg)
What does this mean? (irregular load distribution, strange peeks)
![Page 43: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/43.jpg)
somewhat subtle: unexpected cpu load spikes on utility machines
![Page 44: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/44.jpg)
diagnosis: software release increases cpu load
![Page 45: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/45.jpg)
Confirmation: rollback helps
![Page 46: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/46.jpg)
![Page 47: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/47.jpg)
a little bit about development methods
• requirements gathering (quality varies)• planning• implementation
![Page 48: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/48.jpg)
our implementation languages
• browser– html (+ „furniture graphics“)
– javascript
– css (fairly recently)
– public javascript libraries (fairly recently): MooTools, JQuery
![Page 49: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/49.jpg)
One view of a web application pattern (Application Architecture Guide 2.0: Designing Applications on
the .NET Platform
![Page 50: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/50.jpg)
our implementation languages
• webserver („front end“)– ASP (JScript, mainly 2000 – 2005)
– DotNet 2 (2005 – 2007)
– DotNet 3 (2007 - )
![Page 51: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/51.jpg)
implementation languages
• „middle tier“– purchased DLLs (mostly INBS): 2000 –
– home-built DLL(s) (VBmostly INBS ): ca. 2003 –
– home-built DLLs in DotNet: ca. 2005 -
![Page 52: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/52.jpg)
„back-end“ data base
• SQL• MS SQL add-ons:
– stored procedures
– user defined functions
– triggers
– schedulable DB „jobs“
![Page 53: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/53.jpg)
more „back end“: file servers, application servers, background processes
• various scripting languages– lowly batch files
– „windows script host“
– cygwin (unix-like shells and friends under windows)
– recently some powershell (once called monad)
– windows „scheduled tasks“ (similar to Unix cron)
• special purpose utilities– gathering and digesting log files
– checking synchronization of distributed databases
![Page 54: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/54.jpg)
![Page 55: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/55.jpg)
scaling challenges
• as traffic grows log files get bigger• as a content site ages, content grows• as a site with registered users grows and ages, the number of users
(hopefully increases)
![Page 56: Faz.net technical challenges 2000 - 2009. what i won’t show business numbers, € business plans](https://reader038.vdocuments.mx/reader038/viewer/2022103123/56649d245503460f949fa61b/html5/thumbnails/56.jpg)
ways to scale
• exploit parallelism where possible• buy bigger, faster hardware• make units of work simpler, smaller• other suggestions?• combine these – what works best?