dexador rises
TRANSCRIPT
Dexador Rises
Lisp Meet Up #31 August 26, 2015
Eitaro Fukamachi Somewrite Co., Ltd.
I’m Eitaro Fukamachi @nitro_idiot fukamachi
My Products
• Clack
• Caveman2
• Woo
• quickdocs.org
• Qlot
We Work Remotely
We Work RemotelyWe’re hiring!
Today, let me talk about HTTP client.
HTTP client
Client Server
HTTP client
Client Server
HTTP Request
GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */*
HTTP client
Client Server
HTTP Request
HTTP Response
GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */*
HTTP/1.1 200 OK Date: Mon, 24 Aug 2015 00:41:36 GMT Content-Type: text/html Content-Length: 3771 Connection: keep-alive
HTTP client
Client Server
HTTP Request
HTTP Response
GET / HTTP/1.1 Host: quickdocs.org User-Agent: curl/7.43.0 Accept: */*
HTTP/1.1 200 OK Date: Mon, 24 Aug 2015 00:41:36 GMT Content-Type: text/html Content-Length: 3771 Connection: keep-alive
ex) Google Chrome, curl, Drakma ex) Apache, nginx, Woo
HTTP client libraries
• Drakma
• trivial-http
• Carrier
HTTP client libraries
• Drakma
• De facto. Full-featured.
• trivial-http
• Simple. A few features. No SSL.
• Carrier
• Asynchronous.
HTTP client libraries
• Drakma (usocket)
• De facto. Full-featured.
• trivial-http (usocket)
• Simple. A few features. No SSL.
• Carrier (cl-async/libuv)
• Asynchronous.
HTTP client libraries
• Drakma (usocket)
• De facto. Full-featured.
• trivial-http (usocket)
• Simple. A few features. No SSL.
• Carrier (cl-async/libuv)
• Asynchronous.
75 LIBRARIES
Required by
3 LIBRARIES
None
Drakma
• Ediware
• Since 2006
• Still maintained at GitHub
Mature??? Easy to use??? Fast???
No.
Pitfall 1: Force URL encoding
• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” tag))
tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => PURI:URI-PARSE-ERROR
• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8)))
tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => “common+lisp”
• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8)))
tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => “common+lisp”
tag = “AKB48” => OK tag = “乃木坂46” => "%E4%B9%83%E6%9C%A8%E5%9D%8246"
• Force URL encoding with PURI
Pitfall (1/3) of Drakma
(drakma:http-request (format nil “http://b.hatena.ne.jp/search/tag?q=~A” (drakma:url-encode tag :utf-8)) :preserve-uri t)
tag = “lisp” => OK tag = “scheme” => OK tag = “clojure” => OK tag = “common lisp” => OK
tag = “AKB48” => OK tag = “乃木坂46” => OK
Pitfall 2: Poor language support
• Poor language support with flexi-streams
Pitfall (2/3) of Drakma
(drakma:http-request “http://www.hatena.ne.jp/”)
(drakma:http-request “http://www.google.co.jp/”)
• Poor language support with flexi-streams
Pitfall (2/3) of Drakma
(drakma:http-request “http://www.hatena.ne.jp/”) ;; => body as UTF-8 string
(drakma:http-request “http://www.google.co.jp/”) ;; => body as byte vector
WARNING: Problems determining charset (falling back to binary): :SHIFT_JIS is not known to be a name for an external format.
Pitfall 3: Error handling
• Tend to forget error handling
Pitfall (3/3) of Drakma
(let* ((body (drakma:http-request “http://cliki.net”)) (parsed-html (plump:parse body))) …)
• Tend to forget error handling
Pitfall (3/3) of Drakma
(let* ((body (drakma:http-request “http://cliki.net”)) (parsed-html (plump:parse body))) …)
It fails if the HTTP response code is 4xx or 5xx.
• Tend to forget error handling
Pitfall (3/3) of Drakma
(multiple-value-bind (body status) (drakma:http-request “http://cliki.net”) (unless (= status 200) (error “An HTTP request failed (Code=~D)” status)) (let ((parsed-html (plump:parse body))) …))
Raise an error unless the status is not 200
• Tend to forget error handling
Pitfall (3/3) of Drakma
(multiple-value-bind (body status) (drakma:http-request “http://cliki.net”) (unless (= status 200) (error “An HTTP request failed (Code=~D)” status)) (let ((parsed-html (plump:parse body))) …))
Raise an error unless the status is not 200
Want to retry???
• Tend to forget error handling
Pitfall (3/3) of Drakma
(block nil (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (go retry)) (return body))))
With Auto-Retrying
• Tend to forget error handling
Pitfall (3/3) of Drakma
(block nil (let ((times 5)) (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (when (= times 0) (error "An HTTP request failed. (Code=~D)” status)) (decf times) (go retry)) (return body)))))
Retry only 5 times
Many pitfalls.
Ridiculous.
We just wanted to send an HTTP request.
Dexador changes it.
• Full-featured. usocket based.
• Use fast-http, QURI, Babel, cl-cookie
Dexador: Another choice
Dexador: APIs
(dex:get “http://lisp.org/“) (dex:post “http://lisp.org/“) (dex:head “http://lisp.org/“) (dex:put “http://lisp.org/“) (dex:delete “http://lisp.org/“)
Dexador: Language support
;; Shift_JIS (dex:get “http://www.google.co.jp/“)
;; EUC-JP (dex:get “https://mixi.jp/“)
Dexador: Error handling
(handler-case (dex:get “http://cliki.net/“) (dex:http-request-failed (e) (warn “An HTTP request failed (Code=~D)” (dex:response-status e))))
Dexador: Error handling
(handler-case (dex:get “http://cliki.net/“) (dex:http-request-forbidden () ;; for 403 forbidden ) (dex:http-request-service-unavailable () ;; for 503 service unavailable ) (dex:http-request-failed (e) (warn “An HTTP request failed (Code=~D)” (dex:response-status e))))
Dexador: Error handling
;; Ignore errors and continue (handler-bind ((dex:http-request-failed #'dex:ignore-and-continue)) (dex:get "http://lisp.org"))
Dexador: Auto-Retrying
;; Auto-retry on 503 error (handler-bind ((dex:http-request-service-unavailable #’dex:retry-request)) (dex:get "http://lisp.org"))
Dexador: Auto-Retrying
;; Retry only 5 times (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5))) (dex:get "http://lisp.org"))
Dexador: Auto-Retrying
;; Retry only 5 times at 3-second intervals (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5 :interval 3))) (dex:get "http://lisp.org"))
Dexador: Auto-Retrying
;; Retry only 5 times at 3-second intervals (handler-bind ((dex:http-request-service-unavailable (dex:retry-request 5 :interval 3))) (dex:get "http://lisp.org"))
(block nil (let ((times 5)) (tagbody retry (multiple-value-bind (body status) (drakma:http-request "http://cliki.net/") (unless (= status 200) (when (= times 0) (error "An HTTP request failed. (Code=~D)” status)) (decf times) (sleep 3) (go retry)) (return body)))))
Dexador
Drakma
You may ask…
You may ask…
…is it fast?
BenchmarkSending GET request 30 times to Local Server
(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexadorw/out conneciton-pool
0.024s
BenchmarkSending GET request 30 times to Local Server
(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexadorw/out conneciton-pool
0.013s
0.024s
Sending GET request 30 times to Local Server(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexadorw/out conneciton-pool
0.013s
0.024s
Benchmark
x1.8 faster!
Sending GET request 30 times to Local Server(lower is better)
0
0.009
0.018
0.026
0.035
Drakma Dexadorw/out conneciton-pool
0.013s
0.024s
Benchmark
x1.8 faster!????
How about requesting over network?
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
0.505s
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
0.396s
0.505s
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
0.396s
0.505s x1.2 faster
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
0.396s
0.505s 0.0036 sec/req
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
0.396s
0.505s 0.0036 sec/req
Too trivial improvement…
How come?
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
0.396s
0.505s
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
0.396s
0.505s
Network Latency +
Connection establishment
Network Latency +
Connection establishment
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
0.396s
0.505s
Network Latency +
Connection establishment
Network Latency +
Connection establishment
The largest bottleneck
Differences of Servers/ClientsHTTP Server HTTP Client
C
S
C C
C
S
Can’t help Network Latency and bandwidth.
Can skip connection establishment…?
• Reuse connections once established
(Implicit) Connection-pooling
• Reuse connections once established
(Implicit) Connection-pooling
;; Establish a new connection (dex:get “http://lisp.org/index.html“)
;; Reuse the above connection (dex:get “http://lisp.org/index.html“)
• Reuse connections once established
(Implicit) Connection-pooling
;; Establish a new connection (dex:get “http://lisp.org/index.html“)
;; Reuse the above connection (dex:get “http://lisp.org/index.html“)
0.727 sec
0.380 sec
Benchmark (again)
BenchmarkSending GET request 30 times to Local Server
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexadorw/out conneciton-pool
Dexadorw/ connection-pool
0.013s
0.024s
BenchmarkSending GET request 30 times to Local Server
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexadorw/out conneciton-pool
Dexadorw/ connection-pool
0.005s
0.013s
0.024s
BenchmarkSending GET request 30 times to Local Server
(lower is better)
0
0.006
0.012
0.018
0.024
Drakma Dexadorw/out conneciton-pool
Dexadorw/ connection-pool
0.005s
0.013s
0.024s
x4.8 faster!
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
Dexadorw/ connection-pool
0.396s
0.505s
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
Dexadorw/ connection-pool
0.219s
0.396s
0.505s
BenchmarkSending GET request 30 times to Remote Server
(lower is better)
0
0.15
0.3
0.45
0.6
Drakma Dexadorw/out conneciton-pool
Dexadorw/ connection-pool
0.219s
0.396s
0.505s Still x2.3 faster!
• It’s pretty common to request to the same host multiple times
• Can expect some performance improvements in real applications
In real applications
Status
• Still BETA (v0.9.7)
• Stabilizing. More performance improvements are secondary importance.
• Bug reports are welcome
• Tested with SBCL, CCL, ABCL and ECL on Travis CI
Status
Try the new player and send me a feedback.
Thanks.
EITARO FUKAMACHI 8arrow.org @nitro_idiot fukamachi