scaling with event-based webservers
DESCRIPTION
Slides from my talk at Open Source Days 2011 on Event-based webservers.TRANSCRIPT
Scaling with event–based webserversMorten Siebuhr
Open Source DaysMarch 5th 2011
1/66
whoami(1)
Defining “web–server work”
Killing Apache
Event–based servers
Using Node.js
Questions
2/66
whoami(1)
3/66
Computer Scientist
Distributed Systems / Scientific Computing
(Web–)developer @ One.com
4/66
Web–development @ one.com:
Rich Internet Applicationse–mail, calendar, galleries, . . .
+ 900.000 customers
5/66
6/66
Defining“Web–server work”
7/66
Typical web–server work
Serving static files
Talking to databases
Not doing lots of computations
≈ 100→ 1000 connections served ASAP
8/66
Typical web–server work
Serving static files
Talking to databases
Not doing lots of computations
≈ 100→ 1000 connections served ASAP
8/66
Typical web–server work
Serving static files
Talking to databases
Not doing lots of computations
≈ 100→ 1000 connections served ASAP
8/66
Typical web–server work
Serving static files
Talking to databases
Not doing lots of computations
≈ 100→ 1000 connections served ASAP
8/66
Typical web–server work
Serving static files
Talking to databases
Not doing lots of computations
≈ 100→ 1000 connections served ASAP
8/66
(A)typical web–server work
Persistent connections
=∞ simultaneous connections
≈ 10000 simultaneous connections
(But we don’t care as much about latency. . . )
9/66
(A)typical web–server work
Persistent connections
=∞ simultaneous connections
≈ 10000 simultaneous connections
(But we don’t care as much about latency. . . )
9/66
(A)typical web–server work
Persistent connections
=∞ simultaneous connections
≈ 10000 simultaneous connections
(But we don’t care as much about latency. . . )
9/66
(A)typical web–server work
Persistent connections
=∞ simultaneous connections
≈ 10000 simultaneous connections
(But we don’t care as much about latency. . . )
9/66
(A)typical web–server work
Persistent connections
=∞ simultaneous connections
≈ 10000 simultaneous connections
(But we don’t care as much about latency. . . )
9/66
waiting...
10/66
sleep(1.0)
11/66
12/66
0 s
5 s
10 s
15 s
20 s
25 s
0 100 200 300 400 500 600 700 800 900 1000
Avera
ge r
esponse tim
e
Concurrent requests
Response time w. sleep(1).
WSGI
13/66
0 s
5 s
10 s
15 s
20 s
25 s
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Avera
ge r
esponse tim
e
Concurrent requests
Response time w. sleep(1).
WSGI
14/66
Threading
15/66
Use threads
16/66
Wasted resources!
17/66
Add threads
18/66
0 s
5 s
10 s
15 s
20 s
25 s
0 100 200 300 400 500 600 700 800 900 1000
Avera
ge r
esponse tim
e
Concurrent requests
Response time w. sleep(1).
WSGIApache
19/66
0 s
5 s
10 s
15 s
20 s
25 s
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Avera
ge r
esponse tim
e
Concurrent requests
Response time w. sleep(1).
WSGIApache
20/66
21/66
22/66
23/66
0 MB
250 MB
500 MB
750 MB
1000 MB
1250 MB
1500 MB
0 100 200 300 400 500 600 700 800 900 1000
Max V
irtu
al M
em
ory
Concurrent requests
Memory usage w. sleep(1).
WSGIApache
25/66
0 MB
250 MB
500 MB
750 MB
1000 MB
1250 MB
1500 MB
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Max V
irtu
al M
em
ory
Concurrent requests
Memory usage w. sleep(1).
WSGIApache
26/66
A single connection
=
1 thread
=
1 C–stack & kernel data structures
+
Locking & switching overhead &c. . .
27/66
A single connection
=
1 thread
=
1 C–stack & kernel data structures
+
Locking & switching overhead &c. . .
27/66
A single connection
=
1 thread
=
1 C–stack & kernel data structures
+
Locking & switching overhead &c. . .
27/66
A single connection
=
1 thread
=
1 C–stack & kernel data structures
+
Locking & switching overhead &c. . .
27/66
28/66
Add memory!
Memory is cheap
Except, actually using it isn’t cheap. . .
29/66
Add memory!
Memory is cheap
Except, actually using it isn’t cheap. . .
29/66
Add memory!
Memory is cheap
Except, actually using it isn’t cheap. . .
29/66
Memory Wall
1986→ 2000:
+ 55% CPU speed P/A
+ 10% RAM speed P/A
(Source: http://www.cs.virginia.edu/papers/Hitting_Memory_Wall-wulf94.pdf)
30/66
Memory Wall
1986→ 2000:
+ 55% CPU speed P/A
+ 10% RAM speed P/A
(Source: http://www.cs.virginia.edu/papers/Hitting_Memory_Wall-wulf94.pdf)
30/66
Memory Wall
1986→ 2000:
+ 55% CPU speed P/A
+ 10% RAM speed P/A
(Source: http://www.cs.virginia.edu/papers/Hitting_Memory_Wall-wulf94.pdf)
30/66
31/66
Threads and connections mapped 1:1
We use a lot of threads
Which we don’t have the memory bandwidth to move around.
32/66
Threads and connections mapped 1:1
We use a lot of threads
Which we don’t have the memory bandwidth to move around.
32/66
Threads and connections mapped 1:1
We use a lot of threads
Which we don’t have the memory bandwidth to move around.
32/66
Events!
33/66
waiting...
34/66
35/66
36/66
Event-basics
Event queue
Event loop
“The Event Loop”
37/66
Event-basics
Event queue
Event loop
“The Event Loop”
37/66
Event-basics
Event queue
Event loop
“The Event Loop”
37/66
Event-basics
Event queue
Event loop
“The Event Loop”
37/66
A single connection
=
1 “Event”
=
Some variables & function
=
Hash table & some pointers
+
Queue of events waiting to be processed
38/66
A single connection
=
1 “Event”
=
Some variables & function
=
Hash table & some pointers
+
Queue of events waiting to be processed
38/66
A single connection
=
1 “Event”
=
Some variables & function
=
Hash table & some pointers
+
Queue of events waiting to be processed
38/66
A single connection
=
1 “Event”
=
Some variables & function
=
Hash table & some pointers
+
Queue of events waiting to be processed
38/66
A single connection
=
1 “Event”
=
Some variables & function
=
Hash table & some pointers
+
Queue of events waiting to be processed
38/66
39/66
39/66
39/66
39/66
Why not earlier
Simply not a problem!
Traditional back–end programmers not used to thinking this way
40/66
Why not earlier
Simply not a problem!
Traditional back–end programmers not used to thinking this way
40/66
Why not earlier
Simply not a problem!
Traditional back–end programmers not used to thinking this way
40/66
What we gain
Relatively low memory use
No threading issues
Client-side programmers have no preconditions.
41/66
What we gain
Relatively low memory use
No threading issues
Client-side programmers have no preconditions.
41/66
What we gain
Relatively low memory use
No threading issues
Client-side programmers have no preconditions.
41/66
What we gain
Relatively low memory use
No threading issues
Client-side programmers have no preconditions.
41/66
Node.js
42/66
Node.js is a set of bindings to the V8 JavaScript enginefor scripting network programs.
— Ryan Dahl
43/66
Node.js is a set of bindings to the V8 JavaScript enginefor scripting network programs.
— Ryan Dahl
43/66
JavaScript
Built for event-based programming.
Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days
44/66
JavaScript
Built for event-based programming.
Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days
44/66
JavaScript
Built for event-based programming.
Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days
44/66
JavaScript
Built for event-based programming.
Netscape needed something fast. . .
. . . JavaScript “designed” and implemented in 14 days
44/66
JavaScript
Built for event-based programming.
Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days
44/66
JavaScript
Built for event-based programming.
Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days
44/66
Google V8
JIT’ing JavaScript to native machine code
= Compiling server code
45/66
Google V8
JIT’ing JavaScript to native machine code
= Compiling server code
45/66
Node.js
Non–browser V8–bindings for various libraries
Everything non–blocking
46/66
A minimal webserver in Node.js. . .
// Import HTTP package
var http = require(’http’);
// Set up basic server
var server = http.createServer(function (req , res) {
res.writeHead (200, {’Content -Type’: ’text/plain ’});
res.end(’Hello World\n’);
});
// Start the server
server.listen (8124, "127.0.0.1");
console.log(’Server running at http ://127.0.0.1:8124/ ’);
47/66
. . . with “database”
// Import HTTP package
var http = require(’http’);
// Set up basic server
var server = http.createServer(function (req , res) {
setTimeout(function () {
res.writeHead (200, {’Content -Type’: ’text/plain ’});
res.end(’Hello World\n’);
}, 1000);
});
// Start the server
server.listen (8124, "127.0.0.1");
console.log(’Server running at http ://127.0.0.1:8124/ ’);
48/66
0 s
5 s
10 s
15 s
20 s
25 s
0 100 200 300 400 500 600 700 800 900 1000
Avera
ge r
esponse tim
e
Concurrent requests
Response time w. sleep(1).
WSGIApacheNode.js
49/66
0 s
5 s
10 s
15 s
20 s
25 s
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Avera
ge r
esponse tim
e
Concurrent requests
Response time w. sleep(1).
WSGIApacheNode.js
50/66
0 MB
250 MB
500 MB
750 MB
1000 MB
1250 MB
1500 MB
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Max V
irtu
al M
em
ory
Concurrent requests
Memory usage w. sleep(1).
WSGIApacheNode.js
51/66
Connections in Node.js
=
1 “Event”
=
1 Closure + callback function
=
1 struct + some pointers
+
Event queue operations + function calls
52/66
Connections in Node.js
=
1 “Event”
=
1 Closure + callback function
=
1 struct + some pointers
+
Event queue operations + function calls
52/66
Connections in Node.js
=
1 “Event”
=
1 Closure + callback function
=
1 struct + some pointers
+
Event queue operations + function calls
52/66
Connections in Node.js
=
1 “Event”
=
1 Closure + callback function
=
1 struct + some pointers
+
Event queue operations + function calls
52/66
Connections in Node.js
=
1 “Event”
=
1 Closure + callback function
=
1 struct + some pointers
+
Event queue operations + function calls
52/66
Alternatives
Well–written thread program!
Erlanghttp://www.erlang.org/
Google GOhttp://golang.org/
Lua Event Machinehttps://github.com/esmil/lem
Nginxhttp://wiki.nginx.org/
Varnishhttp://www.varnish-cache.org/
53/66
Alternatives
Well–written thread program!
Erlanghttp://www.erlang.org/
Google GOhttp://golang.org/
Lua Event Machinehttps://github.com/esmil/lem
Nginxhttp://wiki.nginx.org/
Varnishhttp://www.varnish-cache.org/
53/66
Alternatives
Well–written thread program!
Erlanghttp://www.erlang.org/
Google GOhttp://golang.org/
Lua Event Machinehttps://github.com/esmil/lem
Nginxhttp://wiki.nginx.org/
Varnishhttp://www.varnish-cache.org/
53/66
Alternatives
Well–written thread program!
Erlanghttp://www.erlang.org/
Google GOhttp://golang.org/
Lua Event Machinehttps://github.com/esmil/lem
Nginxhttp://wiki.nginx.org/
Varnishhttp://www.varnish-cache.org/
53/66
Alternatives
Well–written thread program!
Erlanghttp://www.erlang.org/
Google GOhttp://golang.org/
Lua Event Machinehttps://github.com/esmil/lem
Nginxhttp://wiki.nginx.org/
Varnishhttp://www.varnish-cache.org/
53/66
Alternatives
Well–written thread program!
Erlanghttp://www.erlang.org/
Google GOhttp://golang.org/
Lua Event Machinehttps://github.com/esmil/lem
Nginxhttp://wiki.nginx.org/
Varnishhttp://www.varnish-cache.org/
53/66
Alternatives
Well–written thread program!
Erlanghttp://www.erlang.org/
Google GOhttp://golang.org/
Lua Event Machinehttps://github.com/esmil/lem
Nginxhttp://wiki.nginx.org/
Varnishhttp://www.varnish-cache.org/
53/66
Working with Node.js
54/66
I’m new to JavaScript!
55/66
56/66
JavaScript WTFs
Always add ;!
Function arguments. . .
this scoped to callee object!
Stacktraces + events sometimes look strange
57/66
Nice code!
def process(user):
user = getAuth(user)
if not user:
return ’Fail’
db = getConnnection(params)
data = db.getUserData(user)
if not data:
return ’Fail’
return data
59/66
Boomerang code
function process(user , callback) {
getAuth(req.username , function (err , user) {
if (err) return callback(err);
getDBConnection(params , function(err , db) {
if (err) return callback(err);
db.getUserData(user , function(err , data) {
if (err) return callback(err);
return callback(null , data);
});
});
});
}
60/66
NPM
61/66
Express.js
var app = require(’express ’).createServer ();
app.get(’/’, function(req , res) {
setTimeout(function () {
res.send(’Hello World ’);
}, 1000);
});
app.listen (3000);
62/66
Express.js - Chat server
var app = require(’express ’).createServer (),
chat = [];
app.get(’/’, function(req , res) {
chat.push(res);
});
app.get(’/:msg’, function(req , res) {
chat.forEach(function (conn) {
conn.write(req.params.msg);
});
});
app.listen (3000);
63/66
Questions
(The end)
64/66
http://nodejs.org/
https://github.com/joyent/node
65/66
function process(user , callback) {
var user = null , db = null;
function done_ () {
if (user && db) {
db.getUserData(user , function(err , data) {
if (err) return callback(err);
return callback(null , data);
});
}
}
getAuth(req.username , function (err , retuser) {
if (err) return callback(err);
user = retuser;
done_();
});
getDBConnection(params , function(err , retdb) {
if (err) return callback(err);
db = retdb;
done_();
});
}
66/66