Download - HAProxy scale out using open source
Scale out using open source
HAProxy - Scale out using open source | by Ingo Walz
2
Why HAProxy?
High availability
Powerful loadbalancer for websites due to its proxy nature
Open Source
Enterprise ready
HAProxy - Scale out using open source | by Ingo Walz
3
Enterprise options
ALOHA HAProxy Loadbalancer Appliance
HAProxy Enterprise Edition - HAPEE
http://www.haproxy.com/
HAProxy - Scale out using open source | by Ingo Walz
4
Who's using it?
http://www.haproxy.org/they-use-it.html
HAProxy - Scale out using open source | by Ingo Walz
5
Featureset
Content switching / filtering Asymetric load balancing Priority activation SSL offloading HTTP compression TCP buffering Priority queue / rate shaping Direct server return (DSR)
http://en.wikipedia.org/wiki/Load_balancing_(computing)#Load_balancer_features
HAProxy - Scale out using open source | by Ingo Walz
6
Looks familiar?
HAProxy - Scale out using open source | by Ingo Walz
7
Fix your Single-Server Environment
Congratulation, your whole environment is one Single Point Of Failure!
HAProxy - Scale out using open source | by Ingo Walz
8
Fix your Single-Server Environment
Always try to follow the principle:
One function per component
Not anymore because it scales the best, more because it's the cleanest way to manage them.
Configure Services, not Servers
HAProxy - Scale out using open source | by Ingo Walz
9
Make your application cluster capable
You need to deploy to a various number of different machines
Do not use something like NFS to workaround
A CI will help you
Session clustering
Avoid to work on the filesystem to save data / user input
Use central technologies to save your data (e.g. databases)
HAProxy - Scale out using open source | by Ingo Walz 10
Installation
HAProxy - Scale out using open source | by Ingo Walz
11
Known procedure
$ wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.6.tar.gz
$ tar xvzf haproxy-1.5.6.tar.gz
$ cd haproxy-1.5.6
$ ./configure USE_OPENSSL=1 USE_PCRE=1
$ make
$ sudo make install
http://www.haproxy.org/#down
HAProxy - Scale out using open source | by Ingo Walz 12
Configuration
HAProxy - Scale out using open source | by Ingo Walz
13
Configuration sections
global# process wide and eventually OS specific
# some have CLI equivalents
[ .. ]
defaults# set default parameters for all following sections
[ .. ]
frontend# describes a set of listening sockets accepting client connections
[ .. ]
backend# describes a set of servers to which the proxy will connect
# to forward incoming connections
[ .. ]
listen# defines a complete proxy with its frontend and backend parts combined in one section.
# It is generally useful for TCP-only traffic
[ .. ]
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
14
TCP vs HTTP loadbalancing
defaults
mode tcp # Can balance everything, the default
defaults
mode http # But you want that!
Layer 7 loadbalancing advantages● Request inspection● Content switching● Header manipulation● Cookie persistence● Advanced health checks
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
15
Loadbalance your nginx
HAProxy - Scale out using open source | by Ingo Walz
16
Loadbalance your nginx
global
maxconn 4096
daemon
defaults
mode http
timeout connect 5000
timeout client 50000
timeout server 50000
frontend www_fe
bind :80
# Close connection to server but keep open for client
option http-server-close
default_backend www
backend www_be
server nginx1 10.0.0.10:80 check
server nginx2 10.0.0.15:80 check
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
17
Still a single point of failure
HAProxy - Scale out using open source | by Ingo Walz
18
HA with HAProxy & keepalived
HAProxy - Scale out using open source | by Ingo Walz
19
HA with HAProxy & keepalived
vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance VIRTUAL {
interface eth0
virtual_router_id 10
state MASTER #state BACKUP
priority 100 #priority 101
advert_int 1
virtual_ipaddress {
10.0.0.30
}
track_script {
chk_haproxy
}
}
/etc/keepalived/keepalived.conf
HAProxy - Scale out using open source | by Ingo Walz
20
HA with public IPs
HAProxy - Scale out using open source | by Ingo Walz
21
HA with public IPs - failover
HAProxy - Scale out using open source | by Ingo Walz 22
The final step
HAProxy - Scale out using open source | by Ingo Walz
23
Loadbalance MySQL - TCP
HAProxy - Scale out using open source | by Ingo Walz
24
Loadbalance MySQL - TCP
frontend mysql_fe
bind :3306
mode tcp
default_backend mysql_be
backend mysql_be
mode tcp
option mysql-check user haproxy
server mysql1 10.0.0.40:3306 check
server mysql2 10.0.0.45:3306 check backup
mysql~> INSERT INTO mysql.user (Host,USER) VALUES ('10.0.0.20','haproxy'); FLUSH PRIVILEGES;
mysql~> INSERT INTO mysql.user (Host,USER) VALUES ('10.0.0.25','haproxy'); FLUSH PRIVILEGES;
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
25
Loadbalance MySQL - TCP
HAProxy - Scale out using open source | by Ingo Walz
26
Loadbalance MySQL - TCP
HAProxy - Scale out using open source | by Ingo Walz 27
All about SSL
HAProxy - Scale out using open source | by Ingo Walz
28
Setup SSL Pass-Through
frontend www_fe
bind :80
bind :443
mode tcp
default_backend www_be
backend www_be
mode tcp
server nginx1 10.0.0.10:443 check
server nginx2 10.0.0.15:443 check
No HTTP mode possible – how to inspect encrypted headers?
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
29
SSL Termination – why you should offload
Single configuration point for all certificates
Certificates not widely spread across the infrastructure
Offload the decryption load
Typically, your HAProxy will have a bit of CPU left
You need to decrypt to inspect the request information
HAProxy - Scale out using open source | by Ingo Walz
30
Setup SSL offloading
frontend www_fe
bind :80
bind :443 ssl crt /etc/haproxy/sample.pem
# Close connection to server but keep open for client
option http-server-close
default_backend www_be
backend www_be
server nginx1 10.0.0.10:80 check
server nginx2 10.0.0.15:80 check
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout \
ssl/sample.key -out ssl/sample.crt
$ cat ssl/sample.key ssl/sample.crt > ssl/sample.pem
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
31
SNI – How it works
● Multiple certificates per IP / frontend profile● Client and server need to support it
HAProxy - Scale out using open source | by Ingo Walz
32
Setup SNI
frontend www_fe
bind :80
# sample.pem as default, other pem's based on host header
bind :443 ssl crt /etc/haproxy/sample.pem crt /etc/haproxy/certs/
# Content switch based on certificate (and based on host)
use_backend sample1 if { ssl_fc_sni sample1 }
use_backend sample2 if { ssl_fc_sni sample2 }
default_backend www_be
backend sample1
server nginx1 10.0.0.10:80 check
backend sample2
server nginx2 10.0.0.15:80 check
backend www_be
server nginx1 10.0.0.10:80 check
server nginx2 10.0.0.15:80 check
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz 33
Secure your entry point
HAProxy - Scale out using open source | by Ingo Walz
34
Protect against syn flooding
# Consider this amount of clients as valid
$ sysctl -w net.ipv4.tcp_max_syn_backlog=”4096”
# Once net.ipv4.tcp_max_syn_backlog is reached, enable syn cookies
$ sysctl -w net.ipv4.tcp_syncookies=1
# Enable reverse path filtering, is the source routable through the incoming interface?
$ sysctl -w net.ipv4.conf.all.rp_filter=1
HAProxy - Scale out using open source | by Ingo Walz
35
Basic iptables
$ cat iptables.sh
#!/bin/bash
iptables -F # Drop current table
# Drop incoming traffic (eth0 is the public available interface)
iptables -i eth0 -P INPUT DROP
iptables -i eth0 -P FORWARD DROP
# Allow outgoing traffic
iptables -P OUTPUT ACCEPT
# Allow ping
iptables -i eth0 -A INPUT -p ICMP -j ACCEPT
# Allow SSH (this should be avoided, SSH to haproxy via internal interface / through VPN)
iptables -i eth0 -A INPUT -j ACCEPT -p tcp --dport 22
# Allow HTTP
iptables -i eth0 -A INPUT -j ACCEPT -p tcp --dport 80
# Allow HTTPS
iptables -i eth0 -A INPUT -j ACCEPT -p tcp --dport 443
# Allow connections from localhost on every port
iptables -i eth0 -A INPUT -j ACCEPT -s 127.0.0.1
# Already opened connections are accepted on every port (required for some daemons)
iptables -i eth0 -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
HAProxy - Scale out using open source | by Ingo Walz
36
Use HAProxy to secure your environment
frontend www_fe
bind :80
bind :443 ssl crt /usr/local/etc/haproxy/sample.pem
option http-server-close
#detect and reject shellshock requests
reqdeny ^[^:]+:\s*\(\s*\)\s+\{
reqdeny ^[^:]+:\s+.*?(<<[^<;]+){5,}
#This rule to display SSLv3 error message
acl sslv3 ssl_fc_protocol SSLv3
http-request allow if sslv3
use_backend backend_sslv3 if sslv3
default_backend www_be
backend backend_sslv3
mode http
errorfile 503 /usr/local/etc/haproxy/pages/poodle.http
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
37
Check SSLv3 error message
$ openssl s_client -connect 10.0.0.30:443 -ssl3
[ … ]
SSL-Session:
Protocol : SSLv3
---
GET /
[ … ]
<html>
<head>
<title>SSLv3 detected</title>
</head>
[ … ]
</html>
HAProxy - Scale out using open source | by Ingo Walz
38
Persistent HTTP loadbalancing
backend www_be
cookie PHPSESSID prefix
server nginx1 10.0.0.10:80 cookie nginx1 minconn 10 maxconn 20 check
server nginx2 10.0.0.15:80 cookie nginx2 minconn 10 maxconn 20 check
# Set-Cookie:PHPSESSID=nginx1~7cmjd41klupaderap0q7tve357; path=/
Persistence only if PHPSESSID cookie is set!
backend www_be
cookie server insert indirect nocache
server nginx1 10.0.0.10:80 cookie nginx1 minconn 10 maxconn 20 check
server nginx2 10.0.0.15:80 cookie nginx2 minconn 10 maxconn 20 check
# Set-Cookie:server=nginx1; path=/
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
39
ACL
Extract data from request / response stream
Perform content switching
Conditional request handling
Can help you to secure your environment
E.g. display an error message for SSLv3
HAProxy - Scale out using open source | by Ingo Walz
40
Loadbalancing algorithms (most useful)
roundrobin
leastconn
Suggested if you have very long sessions
source
Only useful in TCP environments
Other methods availablehttp://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-balance
HAProxy - Scale out using open source | by Ingo Walz
41
“Reverse proxy” usage
frontend www_fe
bind :80
bind :443
[ … ]
use_backend nginx1_backend if { path_beg /nginx1 }
use_backend nginx2_backend if { path_beg /nginx2 }
backend nginx1_backend
reqrep ^([^\ :]*)\ /nginx1(/.*) \1\ /\2
server nginx1 10.0.0.10:80 cookie nginx1 check
backend nginx2_backend
reqrep ^([^\ :]*)\ /nginx2(/.*) \1\ /\2
server nginx2 10.0.0.15:80 cookie nginx1 check
You need to cut nginx1/2 from the request
/usr/local/etc/haproxy/haproxy.cfg
HAProxy - Scale out using open source | by Ingo Walz
42
Statistics
HAProxy - Scale out using open source | by Ingo Walz 43
Run the example
HAProxy - Scale out using open source | by Ingo Walz
44
Reminder
HAProxy - Scale out using open source | by Ingo Walz
45
Used technologies
Vagrant >= 1.5.2
ChefDK >= 0.2.0
Berkshelf
HAProxy - Scale out using open source | by Ingo Walz 46
https://github.com/iwalz/zendcon-haproxy
HAProxy - Scale out using open source | by Ingo Walz
47
Project structure
HAProxy - Scale out using open source | by Ingo Walz
48
Vagrant
$ vagrant plugin install vagrant-omnibus
$ vagrant plugin install vagrant-berkshelf
Omnibus for chef solo support
Berkshelf to manage cookbook dependencies
HAProxy - Scale out using open source | by Ingo Walz
49
Vagrant up
$ git clone https://github.com/iwalz/zendcon-haproxy
$ cd zendcon-haproxy
$ vagrant up haproxy1
$ vagrant up haproxy2
$ vagrant up nginx1
$ vagrant up nginx2
$ vagrant up mysql1
$ vagrant up mysql2
Don't simply use `vagrant up`, the Berkshelf
Dependencies will be messed up
HAProxy - Scale out using open source | by Ingo Walz
50
How to continue
Architecture documentation (outdated, but still useful)
http://www.haproxy.org/download/1.3/doc/architecture.txt
Official documentation
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html
Haproxy.com Blog
http://blog.haproxy.com/
HAProxy - Scale out using open source | by Ingo Walz 51
Questions?