lost and found ( for me ? ): configure transparent load balancing with HA Proxy 1.5.2 on CentOS7

Here are trial and error logs when configuring transparent load balance on HAProxy which is running within KVM.

KVM host : Ubuntu 14.04

HAProxy : CentOS7 running within KVM

Web Server : Ubuntu 14.04 running within KVM

Reference

http://blog.haproxy.com/2013/09/16/howto-transparent-proxying-and-binding-with-haproxy-and-aloha-load-balancer/

# tail -1 /etc/centos-release ; uname -ri

CentOS Linux release 7.0.1406 (Core)

3.10.0-123.13.2.el7.x86_64 x86_64

install haproxy

# yum install haproxy –y

how to start/stop haproxy.

# systemctl status haproxy.service

haproxy.service - HAProxy Load Balancer

Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)

Active: inactive (dead)

# systemctl start haproxy.service

# systemctl status haproxy.service

haproxy.service - HAProxy Load Balancer

Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)

Active: active (running) since Tue 2015-01-10 10:03:50 JST; 2s ago

Main PID: 17296 (haproxy-systemd)

CGroup: /system.slice/haproxy.service

|-17296 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy...

|-17297 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

`-17298 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Jan 10 10:03:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...

Jan 10 10:03:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

Jan 10 10:03:50 localhost.localdomain haproxy-systemd-wrapper[17296]: haproxy-systemd-wrapper: e...

Hint: Some lines were ellipsized, use -l to show in full.

# systemctl stop haproxy.service

[root@localhost ~]#

[root@localhost ~]# systemctl status haproxy.service

haproxy.service - HAProxy Load Balancer

Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)

Active: inactive (dead)

[ configure HTTP load balance without transparent ]

<-- 192.168.122.0 ---------> <- 192.168.100.0 -->

client 122.1 ---- 122.198 ha proxy 100.146 ------- 100.195 server

haproxy.cfg looks like this

# cat /etc/haproxy/haproxy.cfg

backend web-backend

balance roundrobin

mode http

server web1 192.168.100.195:80 check

frontend http

bind *:80

mode http

#use_backend web-backend

default_backend web-backend

restart haproxy

systemctl restart haproxy.service

# systemctl status haproxy.service

haproxy.service - HAProxy Load Balancer

Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)

Active: active (running) since Tue 2015-01-10 10:18:54 JST; 9s ago

Main PID: 17387 (haproxy-systemd)

CGroup: /system.slice/haproxy.service

|-17387 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy...

|-17388 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

`-17389 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Jan 10 10:18:54 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...

Jan 10 10:18:54 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

Jan 10 10:18:54 localhost.localdomain haproxy-systemd-wrapper[17387]

access to the HAProxy’s IP from the client.

# curl http://192.168.122.198

When HAProxy load balanced HTTP traffic to the backend server, source IP is NATed from client’s IP(192.168.122.1) to HA Proxy’s IP ( 192.168.100.146 )

[ configure HTTP transparent load balancing ]

confirm that HAProxy has been compiled with transparent support.

HAProxy shipped with CentOS7 distribution supports transparent proxy, so you do not need to compile HAProxy from the source.

# haproxy -vv

HA-Proxy version 1.5.2 2014/07/12

Build options :

TARGET = linux2628

CPU = generic

CC = gcc

CFLAGS = -O2 -g -fno-strict-aliasing

OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :

maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Built with zlib version : 1.2.7

Compression algorithms supported : identity, deflate, gzip

Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013

Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013

OpenSSL library supports TLS extensions : yes

OpenSSL library supports SNI : yes

OpenSSL library supports prefer-server-ciphers : yes

Built with PCRE version : 8.32 2012-11-30

PCRE library supports JIT : no (USE_PCRE_JIT not set)

Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :

epoll : pref=300, test result OK

poll : pref=200, test result OK

select : pref=150, test result OK

stop HAPRoxy

# systemctl stop haproxy.service

enable ip_forawd and ip_nonlocal_bind so that HAProxy can forward packets and bind IP addresses HA proxy does not have.

[root@localhost ~]#

[root@localhost ~]# echo 1 > /proc/sys/net/ipv4/ip_forward

[root@localhost ~]# echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind

next, configure iptables.

at first, I cleared all of iptables rule sets.

[root@localhost ~]# iptables -F -t mangle

[root@localhost ~]# iptables -F

[root@localhost ~]# iptables -F -t nat

configure iptables and ip.

227 iptables -t mangle -N DIVERT

228 iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT

229 iptables -t mangle -A DIVERT -j MARK --set-mark 1

230 iptables -t mangle -A DIVERT -j ACCEPT

231 ip rule add fwmark 1 lookup 100

232 ip route add local 0.0.0.0/0 dev lo table 100

Here are outputs of iptables and ip route.

[root@localhost ~]# iptables -L -n -t mangle

Chain PREROUTING (policy ACCEPT)

target prot opt source destination

DIVERT tcp -- 0.0.0.0/0 0.0.0.0/0 socket

Chain INPUT (policy ACCEPT)

target prot opt source destination

Chain FORWARD (policy ACCEPT)

target prot opt source destination

Chain OUTPUT (policy ACCEPT)

target prot opt source destination

Chain POSTROUTING (policy ACCEPT)

target prot opt source destination

Chain DIVERT (1 references)

target prot opt source destination

MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK set 0x1

ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

[root@localhost ~]# ip route show

default via 192.168.122.1 dev eth1 proto static metric 1024

192.168.100.0/24 dev ens5 proto kernel scope link src 192.168.100.146

192.168.122.0/24 dev eth1 proto kernel scope link src 192.168.122.198

[root@localhost ~]#

[root@localhost ~]# ip rule show

0: from all lookup local

32765: from all fwmark 0x1 lookup 100

32766: from all lookup main

32767: from all lookup default

[root@localhost ~]#

[root@localhost ~]# ip route show table 100

local default dev lo scope host

[root@localhost ~]#

edit haproxy.cfg

backend web-backend

balance roundrobin

mode http

source 0.0.0.0 usesrc clientip

server web1 192.168.100.195:80 check

frontend http

#bind *:80

bind 192.168.122.198:80 transparent

mode http

#use_backend web-backend

default_backend web-backend

start HAProxy.

can not start HAProxy..

[root@localhost haproxy]# systemctl start haproxy.service

[root@localhost haproxy]#

[root@localhost haproxy]# systemctl status haproxy.service

haproxy.service - HAProxy Load Balancer

Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)

Active: inactive (dead)

Jan 10 10:45:46 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

Jan 10 10:45:46 localhost.localdomain haproxy-systemd-wrapper[17601]: haproxy-systemd-wrapper: e...

Jan 10 10:45:46 localhost.localdomain haproxy-systemd-wrapper[17601]: [ALERT] 012/104546 (17602)...

Jan 10 10:45:46 localhost.localdomain haproxy-systemd-wrapper[17601]: haproxy-systemd-wrapper: e...

Jan 10 10:46:07 localhost.localdomain systemd[1]: Stopped HAProxy Load Balancer.

Jan 10 10:46:10 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...

Jan 10 10:46:10 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

check the syslog.

# cat /var/log/messages

Jan 10 10:46:59 localhost systemd: Starting HAProxy Load Balancer...

Jan 10 10:46:59 localhost systemd: Started HAProxy Load Balancer.

Jan 10 10:46:59 localhost haproxy-systemd-wrapper: [ALERT] 012/104659 (17618) : [/usr/sbin/haproxy.main()] Some configuration options require full privileges, so global.uid cannot be changed.

Jan 10 10:46:59 localhost haproxy-systemd-wrapper: haproxy-systemd-wrapper: exit, haproxy RC=256

edit haproxy.cfg

#user haproxy

#group haproxy

user root

group root

start HAProxy

[root@localhost haproxy]# systemctl start haproxy.service

[root@localhost haproxy]#

[root@localhost haproxy]# systemctl status haproxy.service

haproxy.service - HAProxy Load Balancer

Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)

Active: active (running) since Tue 2015-01-10 10:49:08 JST; 5s ago

Main PID: 17630 (haproxy-systemd)

CGroup: /system.slice/haproxy.service

|-17630 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy...

|-17631 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

`-17632 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Jan 10 10:49:08 localhost.localdomain

access to HAProxy’s IP from the client.

# curl http://192.168.122.198

on the HAProxy.

HAProxy load balanced HTTP traffic to the backend server with client IP.

# tcpdump -n -i ens5 port 80

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on ens5, link-type EN10MB (Ethernet), capture size 65535 bytes

11:43:24.789772 IP 192.168.122.1.50545 > 192.168.100.195.http: Flags [S], seq 4269756824, win 14600, options [mss 1460,sackOK,TS val 336384908 ecr 0,nop,wscale 7], length 0

on the server..

nnn, src IP is 192.168.100.1 which is virbr1 address on the KVM host..

Why??

11:43:59.974689 IP 192.168.100.1.56391 > 192.168.100.195.80: Flags [S], seq 4104026007, win 14600, options [mss 1460,sackOK,TS val 336421984 ecr 0,nop,wscale 7], length 0

HAProxy and the server are running within KVM.

on the KVM host.

# ip addr show virbr1

9: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default

link/ether 52:54:00:62:18:29 brd ff:ff:ff:ff:ff:ff

inet 192.168.100.1/24 brd 192.168.100.255 scope global virbr1

valid_lft forever preferred_lft forever

I guess something is wrong with KVM network.

Seen from the capture data collected on both HAProxy and the Web server, HAProxy load balanced HTTP traffic with client’s IP.

I’m using KVM’s default network ( not openvswitch )

How about using openvswitch for the bridge?

Reference

http://www.opencloudblog.com/?p=177

on the KVM host, define network “ovs-test100” to use openvswitch.

# cat /etc/libvirt/qemu/networks/ovstest-100.xml

<name>ovstest-100</name>

<vlan>

</vlan>

</network>

# virsh net-start ovstest-100

Network ovstest-100 started

# virsh net-autostart ovstest-100

Network ovstest-100 marked as autostarted

# virsh net-dumpxml ovstest-100

<name>ovstest-100</name>

<vlan>

</vlan>

</network>

add a bridge for the network “ovstest-100”.

# ovs-vsctl add-br ovs-100

edit VM’s xml file to use that network. ( virsh edit <VM> )

</interface>

</interface>

start the VM.

# virsh start <VM>

Here is the xml file after booting the VM.

</interface>

<vlan>

</vlan>

</virtualport>

</interface>

on the KVM.

# ovs-vsctl show

6b86c7ef-e656-430e-a09e-e9d3189dec52

Bridge "ovsbr0"

Port "em4"

Interface "em4"

Port "ovsbr0"

Interface "ovsbr0"

type: internal

Bridge "ovs-100"

Port "ovs-100"

Interface "ovs-100"

type: internal

Port "vnet4"

tag: 100

Interface "vnet4"

Port "vnet2"

tag: 100

Interface "vnet2"

ovs_version: "2.0.2"

Here is the network topology.

192.168.122.0/24 10.0.0.0/24

client 122.50 --- 122.198 ha proxy 10.10-------openvswitch-------10.100 server

on the server, add the routing.

# route add -host 192.168.122.50 gw 10.0.0.10

on the client

# curl http://192.168.122.198 --interface 192.168.122.50

capture data on the HAProxy

192.168.122.50 : client IP

10.0.0.100 : server’s IP

Running as user "root" and group "root". This could be dangerous.

Capturing on 'eth1'

[ 1222.334216] device eth1 entered promiscuous mode

1 0.000000 192.168.122.50 -> 10.0.0.100 TCP 74 41493 > 80 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=924748 TSecr=0 WS=128

2 0.000792 10.0.0.100 -> 192.168.122.50 TCP 74 80 > 41493 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=85420 TSecr=924748 WS=128

3 0.000846 192.168.122.50 -> 10.0.0.100 TCP 66 41493 > 80 [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=924749 TSecr=85420

4 0.000892 192.168.122.50 -> 10.0.0.100 HTTP 197 GET / HTTP/1.1

5 0.001013 10.0.0.100 -> 192.168.122.50 TCP 66 80 > 41493 [ACK] Seq=1 Ack=132 Win=30080 Len=0 TSval=85420 TSecr=924749

capture data on the server

14:28:56.561720 IP 192.168.122.50.49794 > 10.0.0.100.80: Flags [S], seq 1346616839, win 14600, options [mss 1460,sackOK,TS val 857775 ecr 0,nop,wscale 7], length 0

14:28:56.561834 IP 10.0.0.100.80 > 192.168.122.50.49794: Flags [S.], seq 1872910538, ack 1346616840, win 28960, options [mss 1460,sackOK,TS val 68668 ecr 857775,nop,wscale 7], length 0

14:28:56.562151 IP 192.168.122.50.49794 > 10.0.0.100.80: Flags [.], ack 1, win 115, options [nop,nop,TS val 857776 ecr 68668], length 0

14:28:56.562167 IP 192.168.122.50.49794 > 10.0.0.100.80: Flags [P.], seq 1:132, ack 1, win 115, options [nop,nop,TS val 857776 ecr 68668], length 131

14:28:56.562212 IP 10.0.0.100.80 > 192.168.122.50.49794: Flags [.], ack 132, win 235, options [nop,nop,TS val 68668 ecr 857776], length 0

I was able to confirm HAProxy’s transparent works well within KVM in combination with openvswitch.

lost and found ( for me ? )

lost and found ( for me ? )

configure transparent load balancing with HA Proxy 1.5.2 on CentOS7

No comments:

Post a Comment