Article is heavily based on the previous one about Setting Up a Load-Balancing LVS (NAT) Cluster with Piranha, but this one aims to show how to configure an LVS cluster with direct routing rather than NAT.
Software
Software used in this article:
- CentOS 6.7
- Piranha 0.8.6
- Ipvsadm 1.26
Direct Routing vs NAT
As we mentioned earlier, a Linux LVS cluster consists of two basic groups: the LVS routers and the real servers. To prevent a single point of failure, each groups will contain two member systems.
As per CentOS documentation, the active router serves two roles in the cluster:
- To balance the load on the real servers,
- To check the integrity of the services on each of the real servers.
The backup router’s job is to monitor the active router and assume its role in the event of failure.
Direct routing allows the real servers to process and route packets directly to requesting users rather than passing all outgoing packets through the LVS router. In this case an LVS router only has to process the client-to-server half of a connection via direct routing, and the response packets can follow separate network routes to the clients. This can increase the scalability of virtual servers and reduce the possibility of network performance issues (as the LVS router only processes incoming packets).
Networking and IP Addresses
Our network is set up as follows:
- 10.10.1.1/24 – LAN with access to the Internet.
Hostnames and roles of the virtual machines we are going to use:
- lvs-d01 – the active LVS router,
- lvs-d02 – the backup LVS router,
- lvs-d03/lvs-d04 – real servers, both running a pre-configured Apache webserver with SSL.
See the schema below for more information.
LVS-D01 and LVS-D02 Nodes’ Network Configuration
Network configuration on the lvs-d01 node:
[root@lvs-d01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 TYPE=Ethernet ONBOOT=yes BOOTPROTO=static PEERDNS=no USERCTL=no IPADDR=10.10.1.21 NETMASK=255.255.255.0 GATEWAY=10.10.1.1
Routing table on the lvs-d01:
[root@lvs-d01 ~]# ip ro 10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.21 169.254.0.0/16 dev eth0 scope link metric 1002 default via 10.10.1.1 dev eth0
Network configuration on the lvs-d02 node:
[root@lvs-d02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 TYPE=Ethernet ONBOOT=yes BOOTPROTO=static PEERDNS=no USERCTL=no IPADDR=10.10.1.22 NETMASK=255.255.255.0 GATEWAY=10.10.1.1
Routing table on the lvs-d02:
[root@lvs-d02 ~]# ip ro 10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.22 169.254.0.0/16 dev eth0 scope link metric 1002 default via 10.10.1.1 dev eth0
LVS-D03 and LVS-D04 Real Servers Nodes’ Network Configuration
Network configuration on the lvs-d03 node:
[root@lvs-d03 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 TYPE=Ethernet ONBOOT=yes BOOTPROTO=static PEERDNS=no USERCTL=no IPADDR=10.10.1.23 NETMASK=255.255.255.0 GATEWAY=10.10.1.1
Routing table on the lvs-d03:
[root@lvs-d03 ~]# ip ro 10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.23 169.254.0.0/16 dev eth0 scope link metric 1002 default via 10.10.1.1 dev eth0
Network configuration on the lvs-d04 node:
[root@lvs-d04 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 TYPE=Ethernet ONBOOT=yes BOOTPROTO=static PEERDNS=no USERCTL=no IPADDR=10.10.1.24 NETMASK=255.255.255.0 GATEWAY=10.10.1.1
Routing table on the lvs-d04:
[root@lvs-d04 ~]# ip ro 10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.24 169.254.0.0/16 dev eth0 scope link metric 1002 default via 10.10.1.1 dev eth0
LVS Router Installation
Configure LVS-D01 Node
Install ipvsadm and Piranha packages:
# yum install -y piranha ipvsadm
Ensure the IP Virtual Server Netfilter kernel module is loaded:
# modprobe ip_vs
Set a password for the Piranha configuration tool:
# /usr/sbin/piranha-passwd
Once the password has been set, start the piranha-gui service:
# /etc/init.d/piranha-gui start
Note that we do not need to start the pulse service until configuration using the Piranha configuration tool is complete.
On the LVS routers, there are four services which need to be set to activate at boot time:
# chkconfig piranha-gui on # chkconfig pulse on # chkconfig iptables on # chkconfig sshd on
Iptables service is required for multi-port services and firewall marks. SSH service is for remote logins and should be self explanatory I think.
Add a firewall rule to allow HTTP access to Piranha WebUI:
# iptables -A INPUT -s 10.10.1.0/24 -p tcp --dport 3636 -j ACCEPT
Add a rule for traffic that we’re balancing (HTTP and HTTPS):
# iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT
Configure LVS-D02 Node
Install ipvsadm:
# yum install -y ipvsadm
Enable services:
# chkconfig pulse on # chkconfig iptables on # chkconfig sshd on
Copy sysctl.conf from the lvs-d01 node:
[root@lvs-d01 ~]# scp /etc/sysctl.conf root@lvs-d02:/etc/
Add a rule for traffic that we’re balancing (HTTP and HTTPS):
# iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT
When finished configuring direct routing with Piranha, copy the lvs.cf file from the lvs-01 node:
[root@lvs-d01 ~]# scp /etc/sysconfig/ha/lvs.cf root@lvs-d02:/etc/sysconfig/ha/
Multi-Port Services
We are going to use multi-port services (HTTP and HTTPS), therefore firewall marks to bundle together different, but related protocols, are required. However, the job of assigning firewall marks must be performed by the network packet filter, iptables, outside of Piranha configuration tool.
Assigning firewall marks on both routers, the lvs-d01 and the lvs-d02:
# iptables -t mangle -A PREROUTING -p tcp -d 10.10.1.20/32 -m multiport --dport 80,443 -j MARK --set-mark 80
Where 10.10.1.20 is our virtual IP address. Save iptables rules so that they get restored after a reboot:
# service iptables save
Direct Routing Real Server Node Configuration
On each real server node (lvs-d03 and lvs-d04), run the following command for the VIP 10.10.1.20 and protocol combination intended to be serviced for the real server:
# iptables -t nat -A PREROUTING -p tcp -d 10.10.1.20 -m multiport --dport 80,443 -j REDIRECT
The command above will cause the real servers to process packets destined for the VIP and port that they are given. Make sure that firewall changes are saved and restored after a restart:
# service iptables save
LVS Direct Routing Configuration with Piranha
Navigate a web browser to http://10.10.1.31:3636/.
Configure the primary server and pick a direct network type (should be the default option).
Add and configure a virtual server. 10.10.1.20 is our virtual IP address. Firewall mark is used to bundle HTTP connections on port 80 and HTTPS connections on port 443.
Persistence is like a timer. It is set to 30 seconds. When a client connects to a service, LVS remembers the last connection for 30s.
Add and activate the real servers.
Configure redundancy by adding the lvs-d02 node’s IP address.
Now, when everything is configured and saved, start the pulse service on the lvs-d01 node:
# service pulse start
At this point we should be able to browse the virtual IP 10.10.1.20 via both HTTP and HTTPS.
Start the pulse service on the lvs-d02 node. It should act as a backup node.
Check LVS status on the active lvs-d01 node:
# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 80 wlc persistent 30 -> 10.10.1.23:80 Route 1 0 0 -> 10.10.1.24:80 Route 1 0 0
lvs.cf
Content of the /etc/sysconfig/ha/lvs.cf is shown below for referencing purposes.
[root@lvs-d01 ~]# cat /etc/sysconfig/ha/lvs.cf
serial_no = 5
primary = 10.10.1.21
service = lvs
backup_active = 1
backup = 10.10.1.22
heartbeat = 1
heartbeat_port = 539
keepalive = 2
deadtime = 6
network = direct
debug_level = NONE
monitor_links = 1
syncdaemon = 0
virtual http_server {
active = 1
address = 10.10.1.20 eth0:1
vip_nmask = 255.255.255.255
fwmark = 80
port = 80
persistent = 30
send = "GET / HTTP/1.0\r\n\r\n"
expect = "HTTP"
use_regex = 0
load_monitor = none
scheduler = wlc
protocol = tcp
timeout = 6
reentry = 15
quiesce_server = 0
server lvs-d03 {
address = 10.10.1.23
active = 1
weight = 1
}
server lvs-d04 {
address = 10.10.1.24
active = 1
weight = 1
}
}
References
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Load_Balancer_Administration/index.html