you’re in a defense talk · running at 99%: surviving an application dos ryan huber @ryanhuber...
TRANSCRIPT
![Page 1: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/1.jpg)
Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber [email protected] !1
![Page 2: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/2.jpg)
$bad news
you’re in a DEFENSE talk
![Page 3: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/3.jpg)
$whoami
ryan huber
risk.io orbitz.com
ebookers.com
small local ISP
![Page 4: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/4.jpg)
$contest
ssid: DoS2own
key: DoS2own!
target: http://feeblechat.com/ or http://10.0.0.2
rules: network DoS doesn’t count, MITM meh..
goal: polling must fail for > 30 seconds
prize: MY WEBSERVER!
![Page 5: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/5.jpg)
$topics
app DoS intro/attack demo
mitigation strategies
bouncer
recap
![Page 6: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/6.jpg)
$99%?
service 99% of users (limit false positives)
at 99% utilization (know your capacity)
![Page 7: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/7.jpg)
$denial of service
network DoS application DoS
![Page 8: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/8.jpg)
$topics
app DoS primer/attack demo
mitigation strategies
bouncer
recap
![Page 9: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/9.jpg)
$application DoSits goal is
goal is to exceed your capacity to handle requests it is
effective with few attack resources
targets or creates slow operations
more difficult to detect
no easy off-the-shelf mitigation
![Page 10: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/10.jpg)
$app DoS categories
webserver
your application
![Page 11: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/11.jpg)
$apache
the number we care about is
MaxClients
![Page 12: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/12.jpg)
$slow headers
(slow loris)
connect
send a header every X seconds
wash, rinse, repeat
![Page 13: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/13.jpg)
$slow POST
(R U Dead Yet)
find a form
POST 2gb of data @ .0000001KB/s
![Page 14: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/14.jpg)
$slow read
GET /a_page
read it @ .000000001KB/s
![Page 15: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/15.jpg)
$webserver DoS demo
![Page 16: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/16.jpg)
$app DoS (your_app)
(targeted attack)
repeatedly execute expensive queries
large downloads
exceed a backend connection pool
create many sessions
![Page 17: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/17.jpg)
$topics
app DoS primer/attack demo
mitigation strategies
bouncer
recap
![Page 18: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/18.jpg)
$apache
mpm
mod_security (Sec{Read,Write}StateLimit)
mod_reqtimeout/mod_qos
varnish/reverse proxy
![Page 19: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/19.jpg)
$strategies
keep slow pages behind login
limit POSTs
don’t generate sessions on GET /
leverage a CDN for large/static content
change webserver software
¡optimize your code!
![Page 20: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/20.jpg)
$identification
user-agent
same url
no referer
hidden link(s)
geoip
ignores cookies/session
missing common headers
request timing
first seen time
proof of work
If-Modified-Since?
last resort: captcha
![Page 21: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/21.jpg)
$a real example
german website
100,000 hosts
3 req/min
random valid URLs
![Page 22: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/22.jpg)
$topics
app DoS primer/attack demo
mitigation strategies
bouncer
recap
![Page 23: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/23.jpg)
$bouncer
(written in node)
(inspired by netflow)
![Page 24: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/24.jpg)
$why node.js?!
because it’s @hipsterhacker approved!
![Page 25: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/25.jpg)
$seriously why node.js?
asynchronous
fast when your task is not CPU bound
great lib node-http-proxy
well known language
JSON is native
![Page 26: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/26.jpg)
$goals
minimal code
small memory footprint
fail open
works in cloud
JSON messaging
decisions made outside of proxy code
![Page 27: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/27.jpg)
$architecture
GRAPHIC HERE (combine next slide)
![Page 28: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/28.jpg)
$message format{"time":1379603264938,"type":"connect",!"host":"10.0.0.150"}!!{"time":1379603264940,"type":"request",!"host":"10.0.0.150",!"url":"/changelog/","method":"GET",!"headers": (....),"uuid":!"f42095a1-3a4b-41fc-b005-46f504cde2a0"}!!
{"time":1379603263662,"type":"end",!"uuid":!"f42095a1-3a4b-41fc-b005-46f504cde2a0"}
![Page 29: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/29.jpg)
$proxy.js
reverse proxy
has own blacklist
has own greylist
has own disabled URL list
236 lines of code
dynamic header and request timeouts
![Page 30: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/30.jpg)
$proxy.js (does):
closes blacklisted sockets immediately
monitors total time to send headers (mitigate slow loris)
monitors total time from request to end of response
assigns UUID (v4) to every request
forwards request records to aggregator
![Page 31: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/31.jpg)
$aggregator.js (does)
links proxies and consumers
multiplex events to consumers
multiplex commands to proxies
64 lines of code
![Page 32: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/32.jpg)
$consumers
you write these
“drink from the (JSON) firehose”
make independent decisions
can be sized to the problem
send commands upstream via aggregator
![Page 33: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/33.jpg)
$consumer commands
block/unblock
grey
durl/eurl
htimeout
rtimeout
flush
![Page 34: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/34.jpg)
$example commands
BLOCK 10.0.0.1|10000!!DURL /puppies_in_santa_costume.jpg!GREY 192.168.1.1|60000!!HTIMEOUT 2000!RTIMEOUT 10000
![Page 35: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/35.jpg)
$example consumer 1
track TCP connection attempts over a time period
cross the threshold, block
![Page 36: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/36.jpg)
$demo consumer 1
![Page 37: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/37.jpg)
$example consumer 2
track pages using the most time
track hosts using those pages
disable the url for hosts that cross threshold
![Page 38: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/38.jpg)
$demo consumer 2
![Page 39: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/39.jpg)
$demo consumer 2a
thi
![Page 40: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/40.jpg)
$example consumer 3
python + redis
sorted sets are AWESOME
times stored for each ip with specified granularity
![Page 41: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/41.jpg)
$demo consumer 3
![Page 42: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/42.jpg)
$logs
echo -e ‘C\n’ | ncat (aggregator) 5555 | gzip > /tmp/meh.log
![Page 43: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/43.jpg)
$running
proxy MUST be run with ulimit -n increase
node ‘forever’ for daemonizing
clock sync VERY IMPORTANT
![Page 44: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/44.jpg)
$performance testing!
AWS c1.medium example 2 -> 62k requests/s (datatest.py generated data)
network saturated before CPU
![Page 45: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/45.jpg)
$other uses
weathering a popularity storm
scraper-pocalypse
![Page 46: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/46.jpg)
$the future, Conan?gzip
operationalize
document
amazon amis
library of consumers
¿multicast?
¿log destroyed connections?
![Page 47: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/47.jpg)
$takeaways
DoS mitigation is easier with a complete picture
suggestions are VERY welcome
contribution much appreciated
![Page 49: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/49.jpg)
$thanks!
https://www.github.com/rawdigits/Bouncer
@ryanhuber
![Page 50: you’re in a DEFENSE talk · Running at 99%: Surviving an Application DoS Ryan Huber @ryanhuber rhuber@gmail.com !1](https://reader033.vdocuments.mx/reader033/viewer/2022050611/5fb1ef03391437282e6ed65e/html5/thumbnails/50.jpg)
$DEMO!!11!1!