fatlan/haproxy-keepalived-sec-highloads✨ HAProxy ve Keepalived konusunu load balancer ve cluster'a ek olarak güvenlik(zayıf SSL/Kripto Kullanımı (LOGJAM) (CVE-2015-4000) zafiyeti önlemi) ve yüksek yüklere karşı ele alır.
HAProxy is an open-source software that provides a proxy server service for TCP and HTTP based applications with high availability load balancing.
Keepalived is an open-source software capable of providing IP failover capability for more than two servers. Keepalived uses multicast communication technique among itself.
In our setup, we will use HAProxy for load balancing and Keepalived for IP failover, i.e., to make our HAProxy setup into a Cluster.
Our scenario consists of 3 servers. We will install HAProxy on these 3 servers and make them a load balancer. Then, by installing the Keepalived service, when one of the servers goes down, we will perform IP failover to switch to another server without interruption and keep our load balancer service running.
We will use 4 IP addresses (IPs are completely fictitious)
Now let's install HAProxy and Keepalived services on all 3 servers as follows.
sudo add-apt-repository ppa:vbernat/haproxy-2.7 -y
sudo apt update
sudo apt install haproxy keepalived -y
sudo openssl dhparam -out /etc/haproxy/dhparams.pem 2048
Note1: The servers must have “net.ipv4.ip_nonlocal_bind=1”. Otherwise, the configuration to be used for HAProxy will give a bind error because it cannot host the same IP at the same time and the service will not work. You need to follow the steps below for this.
First, edit the file “vi /etc/sysctl.conf” and paste the following parameter, save and exit.
net.ipv4.ip_nonlocal_bind=1
Then run the following command.
sysctl -p
Note2: If you are going to use 443 SSL, you need to have the Bundle (.crt,.ca,*.key) of your SSLs named “haproxy.pem” in the “/etc/ssl/private/” directory.
Now let's look at the HAProxy configuration as an example below. You will use the same configuration for each HAProxy in your setup.
For this, you will edit the file “/etc/haproxy/haproxy.cfg”. The configuration below supports min HAv2.
vi /etc/haproxy/haproxy.cfg
The default values are also present below
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# maxconn 100000 #replaceable
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
#ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
ssl-dh-param-file /etc/haproxy/dhparams.pem
#tune.ssl.default-dh-param 2048
# nbproc 1
# nbthread 8
tune.maxrewrite 16384
tune.bufsize 32768
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
# maxconn 1000000 #replaceable
timeout connect 3000000
timeout client 6000000
timeout server 6000000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
Dashboard configuration section for HAProxy (prometheus integration added for monitoring)
listen stats
bind fatlan.com:8989
mode http
stats enable
stats uri /stats
option http-use-htx
http-request use-service prometheus-exporter if { path /metrics }
#stats hide-version
stats realm HAProxy\ Statistics
stats auth admin:admin
In the configuration below, ACL is used in two blocks. The first block will run the middleware-fatlan-backend block if the word “rest” appears anywhere in the link, the second block will run the backend fatlan-forum-backend for a different domain request (forum.fatlan.com), otherwise all requests will run in the fatlan-backend443 block. Except for other port redirects.
Redirecting port 80 to port 443 section
frontend fatlan80
bind fatlan.com:80
mode http
redirect scheme https if !{ ssl_fc }
frontend fatlan443
bind fatlan.com:443 ssl crt /etc/ssl/private/haproxy.pem ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
option httplog
option forwardfor
#http-request set-header X-Client-IP req.hdr_ip([X-Forwarded-For])
#option forwardfor except 127.0.0.0/8
option http-server-close
http-request set-header X-Forwarded-Proto https
# reqadd X-Forwarded-Proto:\ https #old config
mode http
default_backend fatlan-backend443
ACL example configuration (proxypass) redirects if the word rest appears anywhere in the link
acl middleware-fatlan path_beg /rest
use_backend middleware-fatlan-backend if middleware-fatlan
ACL for different forum host redirect
acl host_fatlanforum hdr(host) -i forum.fatlan.com
use_backend fatlan-forum-backend if host_fatlanforum
Default redirection part, redirection part for 443
backend fatlan-backend443
mode http
balance roundrobin
stick store-request src
stick-table type ip size 256k expire 30m
option forwardfor
# option httplog
option httpchk HEAD /
server frontend_01 10.10.37.12:8001 check port 8001 inter 3000 rise 2 fall 3
server frontend_02 10.10.37.13:8001 check port 8001 inter 3000 rise 2 fall 3
ACL incoming, rest redirection part
backend middleware-fatlan-backend
mode http
balance roundrobin
stick store-request src
stick-table type ip size 256k expire 30m
option forwardfor
# option httplog
option httpchk OPTIONS /login HTTP/1.0
http-check expect status 200
http-request replace-path (.*)(?:rest\/)(.*) \1\2
# reqrep ^([^\ :]*)\ /rest[/]?(.*) \1\ //\2 #old config
server middleware_01 10.10.37.34:3000 check port 4000 inter 12000 rise 3 fall 3
server middleware_02 10.10.37.35:3000 check port 4000 inter 12000 rise 3 fall 3
ACL incoming, forum redirection part
backend fatlan-forum-backend
mode http
option forwardfor
# option httplog
option httpchk HEAD /
server forum_01 10.10.37.45:8080 check port 8080 inter 3000 rise 2 fall 3
External examples can also be configured as follows.
Example for port 5000;
frontend Panel5000
bind fatlan.com:5000
# option httplog
option forwardfor except 127.0.0.0/8
#option http-server-close
http-request set-header X-Forwarded-Proto https
# reqadd X-Forwarded-Proto:\ https #old config
mode http
default_backend panel-backend5000
Redirection part for port 5000
backend panel-backend5000
mode http
balance roundrobin
stick store-request src
stick-table type ip size 256k expire 30m
option forwardfor
# option httplog
option httpchk HEAD /
server panel_01 10.10.37.43:5000 check port 5000 inter 12000 rise 3 fall 3
server panel_02 10.10.37.44:5000 check port 5000 inter 12000 rise 3 fall 3
Example for 3306 mysql;
frontend fatlanmysql
bind fatlan.com:3306
mode tcp
default_backend fatlanmysql-backend3306
Redirection part for port 3306
backend fatlanmysql-backend3306
mode tcp
server mysql_01 10.10.37.60:3306 check
server mysql_02 10.10.37.61:3306 check backup
server mysql_03 10.10.37.62:3306 check backup
You can also redirect for other domains, subdomains or ports via the same haproxy. Although we already saw the ACL redirect for subdomain above.
Example for subdomain egitim.fatlan.com on port 4444;
frontend egitim4444
bind egitim.fatlan.com:4444
# option httplog
option forwardfor except 127.0.0.0/8
#option http-server-close
http-request set-header X-Forwarded-Proto https
# reqadd X-Forwarded-Proto:\ https #old config
mode http
default_backend egitim-backend4444
Redirection part for subdomain egitim.fatlan.com port 4444
backend egitim-backend4444
mode http
balance roundrobin
stick store-request src
stick-table type ip size 256k expire 30m
option forwardfor
# option httplog
option httpchk HEAD /
server egitim_01 10.10.37.77:4444 check port 4444 inter 12000 rise 3 fall 3
server egitim_02 10.10.37.78:4444 check port 4444 inter 12000 rise 3 fall 3
I have mentioned the example HAProxy configurations above, I made my own configuration and pasted the same configurations on 3 servers.
Now let's configure Keepalived. For Keepalived, there will be partially different parametric settings on 3 servers. For this, we will create and configure the file “/etc/keepalived/keepalived.conf”. By the way, the one with higher “priority” is preferred.
Note1: Keepalived communicates multicast with its other peers and determines master/backup via this method.
You can also capture multicast communication between peers with Tcpdump (tcpdump -n "multicast").
Server 1 (HAProxy+Keepalived)
vrrp_sync_group haproxy {
group {
VI_01
}
}
vrrp_script haproxy_check_script {
script "killall -0 haproxy"
interval 2 # checking every 2 seconds (default: 5 seconds)
fall 3 # require 3 failures for KO (default: 3)
rise 6 # require 6 successes for OK (default: 6)
}
#Virtual interface
vrrp_instance VI_01 {
state MASTER
interface ens3
### Change the 61 id, it will be the same on other peers
virtual_router_id 61
### Change the 103 id, it will be decreasing on other peers
priority 103
authentication {
auth_type PASS
auth_pass 123456
}
# Virtual ip address – floating ip
virtual_ipaddress {
10.10.5.5
}
track_script {
haproxy_check_script
}
}
Server 2 (HAProxy+Keepalived) (I only write the differences)
state BACKUP
priority 102
Server 3 (HAProxy+Keepalived) (I only write the differences)
state BACKUP
priority 101
Note2: If multicast communication between servers is not possible (e.g. KVM, cloud environments, etc.), you need to configure using unicast communication, otherwise it will not work.
Unicast Config: can be added between the configuration as follows.
### For each server, the other peer IP(s) must be specified (written)
...
#Virtual interface
...
unicast_peer {
<anaother_peer_ip>
<anaother_peer_ip>
}
# Virtual ip address – floating ip
...
That's it for the configurations. On all servers, HAProxy should be running without any issues, and so should the keepalived service. In case of any interruption on servers or services, the load balancer will continue to serve from the other running server.
HAProxy and Linux kernel tuning for high loads
sudo vi /etc/security/limits.conf
* soft no le 1000000
* hard no le 1000000
root soft no le 1000000
root hard no le 1000000
sudo vi /etc/default/haproxy
ulimit 1000000
sudo vi /lib/systemd/system/haproxy.service
LimitNOFILE=1000000
sudo vi /etc/sysctl.conf
net.ipv4.ip_local_port_range=1024 65535
net.ipv4.tcp_max_syn_backlog = 100000
net.core.somaxconn = 100000
net.core.netdev_max_backlog = 100000
sudo vi /etc/haproxy/haproxy.cfg
global
nbproc 1
nbthread 8
tune.maxrewrite 16384
tune.bufsize 32768
maxconn 1000000
tune.ssl.cachesize 1000000
defaults
maxconn 1000000
Then reboot, if reboot is not immediately possible
sudo systemctl daemon-reload
sudo systemctl restart haproxy.service
Also for testing weak SSL/Crypto usage (LOGJAM) (CVE-2015-4000) you can use the following command or https://www.ssllabs.com/ssltest/
sudo nmap -sV --script ssl-enum-ciphers -p 443 fatlan.com
sudo nmap -p 443 --script ssl-cert fatlan.com
openssl s_client -connect fatlan.com:443
Also, in the configs above, you can use tcpdump to capture the client IP via the x-forwarder-for (option forwardfor) configuration, i.e., to verify and capture IPs on the haproxy side.
sudo tcpdump -i ens3 -A -s 10240 | grep -v IP | egrep --line-buffered "..(GET |\.HTTP\/|POST |HEAD )|^[A-Za-z0-9-]+: " |sed -r 's/..(GET |HTTP\/|POST |HEAD )/\n\n\1/g'
