Lvs + keepalived + nginx 高可用負載均衡

https://www.cnblogs.com/liuyisai/p/5990645.html

lvs+keepalived+nginx實作高性能負載均衡叢集

一、為什麼要使用負載均衡技術？

1、系統高可用性

2、  系統可擴充性

3、  負載均衡能力

LVS+keepalived能很好的實作以上的要求，LVS提供負載均衡，keepalived提供健康檢查，故障轉移，提高系統的可用性！采用這樣的架構以後很容易對現有系統進行擴充，隻要在後端添加或者減少realserver，隻要更改lvs的配置檔案，并能實作無縫配置變更！

二、LVS+Keepalived介紹

1、  LVS

LVS是一個開源的軟體，可以實作LINUX平台下的簡單負載均衡。LVS是Linux Virtual Server的縮寫，意思是Linux虛拟伺服器。目前有三種IP負載均衡技術（VS/NAT、VS/TUN和VS/DR）；八種排程算法（rr,wrr,lc,wlc,lblc,lblcr,dh,sh）。

2、  keepalived

Keepalived 是運作在lvs 之上，它的主要功能是實作真實機的故障隔離及負載均衡器間的失敗切換，提高系統的可用性

三、環境：

四台伺服器，系統全為CentOS6.8：

192.168.2.203 master lvs+keepalived

192.168.2.202 backup lvs+keepalived

192.168.2.204 web1（nginx）

192.168.2.205 web2 （nginx）

vip：192.168.2.13

其中nginx已預裝好，這裡不再寫搭建過程

四、搭建并配置

1、分别在backup lvs和master lvs上安裝lvs

1root@bogon src]# yum -y install ipvsadm 2已加載插件：fastestmirror 3設定安裝程序 4Determining fastest mirrors 5epel/metalink                                                                                  |5.4kB00:00 6*base: mirror.lzu.edu.cn 7... ... 8已安裝: 9ipvsadm.x86_640:1.26-4.el6                                                                                          1011作為依賴被安裝:12libnl.x86_640:1.1.4-2.el6                                                                                          1314完畢！

2、把ipvsadm子產品加載進系統

1[root@bogon src]# ipvsadm2IP Virtual Server version1.2.1(size=4096)3Prot LocalAddress:Port Scheduler Flags4-> RemoteAddress:Port          Forward Weight ActiveConn InActConn5[root@bogon src]#lsmod|grep ip_vs6ip_vs12689707libcrc32c12461 ip_vs8ipv6336282270ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6

3、分别在backup lvs和master lvs上安裝keepalived（keepalived官網：http://www.keepalived.org/）

[root@bogon src]#tarzxf keepalived-1.2.24.tar.gz

[root@bogon src]# cd keepalived-1.2.24[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/報錯：

configure: error:

  !!! OpenSSL is not properly installed on your system. !!!  !!! Can not include OpenSSL headers files.            !!!

系統缺少openssl-devel包所緻

安裝openssl-devel

root@bogon keepalived-1.2.24]#yum-yinstallopenssl-devel

再次編輯安裝

[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/Keepalived configuration------------------------Keepalived version      : 1.2.24Compiler                : gccPreprocessor flags      : -I/lib/modules/2.6.32-642.3.1.el6.x86_64//includeCompiler flags          : -Wall -Wunused -Wstrict-prototypes

Linker flags            :

Extra Lib                : -ldl -lssl -lcrypto

Use IPVS Framework      : Yes

IPVS use libnl          : No

IPVS syncd attributes    : No

IPVS 64 bit stats        : No

fwmark socket support    : Yes

Use VRRP Framework      : Yes

Use VRRP VMAC            : Yes

Use VRRP authentication  : Yes

With ip rules/routes    : Yes

SNMP keepalived support  : No

SNMP checker support    : No

SNMP RFCv2 support      : No

SNMP RFCv3 support      : No

SHA1 support            : No

Use Debug flags          : No

Stacktrace support      : No

Memory alloc check      : No

libnl version            : None

Use IPv4 devconf        : No

Use libiptc              : No

Use libipset            : No

Build genhash            : Yes

Build documentation      : No

[root@bogon keepalived-1.2.24]#make&&makeinstall

[root@bogon keepalived-1.2.24]#ln-s /usr/local/sbin/keepalived /sbin/[root@bogon keepalived-1.2.24]# chkconfig --add keepalived

[root@bogon keepalived-1.2.24]# chkconfig --level35keepalived on

4、配置keepalived

lvs-master的配置檔案如下

[root@bogon keepalived-1.2.24]#cat/etc/keepalived/keepalived.conf! Configuration Fileforkeepalivedglobal_defs {　　　　　　#全局配置部分#  notification_email {#email 通知，基本不用此處是以注釋掉#    [email protected]#    [email protected]#    [email protected]#  }#  notification_email_from [email protected]#  smtp_server192.168.200.1#  smtp_connect_timeout 30router_id LVS_DEVEL#設定lvs的id，在一個網絡内應該是唯一的vrrp_skip_check_adv_addr　　  vrrp_strict  vrrp_garp_interval0  vrrp_gna_interval 0}vrrp_instance VI_1 {　　　　#vrrp執行個體定義部分    state MASTER#設定lvs的狀态，報錯MASTER和BACKUP兩種，必須大寫interface eth1#設定對外服務的接口virtual_router_id60　　#設定虛拟路由标示，這個标示是一個數字，同一個vrrp執行個體使用唯一标示    priority 100　　　　#定義優先級，數字越大優先級越高，在一個vrrp——instance下，master的優先級必須大于backup    advert_int 1　　　　#設定master與backup負載均衡器之間同步檢查的時間間隔，機關是秒    authentication {　　#設定驗證類型和密碼

        auth_type PASS　　#主要有PASS和AH兩種

        auth_pass 1111　　#驗證密碼，同一個vrrp_instance下MASTER和BACKUP密碼必須相同    }

    virtual_ipaddress {　　#設定虛拟ip位址，可以設定多個，每行一個

        192.168.2.13    }

}

virtual_server 192.168.2.1380 {　　#設定虛拟伺服器，需要指定虛拟ip和服務端口

    delay_loop 3#健康檢查時間間隔lb_algo rr #負載均衡排程算法lb_kind DR#負載均衡轉發規則persistence_timeout 50#設定會話保持時間，對動态網頁非常有用    protocol TCP　　　　#指定轉發協定類型，有TCP和UDP兩種

    real_server 192.168.2.20480 {　　#配置伺服器節點1，需要指定real server的真實IP位址和端口

        weight 1　　　　#設定權重，數字越大權重越高    TCP_CHECK { 　　　　#realserver的狀态監測設定部分機關秒

            connect_timeout 3　　　　#逾時時間            nb_get_retry 3　　　　　　#重試次數            delay_before_retry 3　　　　#重試間隔        connect_port 80    　　#監測端口

    }

    real_server 192.168.2.20580 {

    weight 1    TCP_CHECK {

    connect_timeout 3    nb_get_retry 3    delay_before_retry 3    connect_port 80        }

LVS-backup的配置檔案如下

[root@bogon keepalived-1.2.24]#cat/etc/keepalived/keepalived.conf! Configuration Filefor keepalived

global_defs {

#  notification_email {

#    [email protected]

#    [email protected]

#    [email protected]

#  }

#  notification_email_from [email protected]

#  smtp_server 192.168.200.1  smtp_connect_timeout 30  router_id LVS_DEVEL

  vrrp_skip_check_adv_addr

  vrrp_strict

  vrrp_garp_interval 0  vrrp_gna_interval 0}

vrrp_instance VI_1 {

    state BACKUP

    interface eth1

    virtual_router_id 60    priority 80    advert_int 1    authentication {

        auth_type PASS

        auth_pass 1111    }

    virtual_ipaddress {

virtual_server 192.168.2.1380 {

    delay_loop 3    lb_algo rr

    lb_kind DR

    persistence_timeout 3    protocol TCP

    real_server 192.168.2.20480 {

        weight 1    TCP_CHECK {

            connect_timeout 3            nb_get_retry 3            delay_before_retry 3        connect_port 80   

5、realserver的配置

兩台web伺服器都要執行下面腳本

[root@bogon www]#cat/etc/rc.d/init.d/realserver.sh

#!/bin/bash

# description: Config realserver lo and apply noarp

SNS_VIP=192.168.2.13/etc/rc.d/init.d/functions

case"$1"instart)

      ifconfiglo:0$SNS_VIP netmask255.255.255.255 broadcast $SNS_VIP

      /sbin/route add -host $SNS_VIP dev lo:0echo"1">/proc/sys/net/ipv4/conf/lo/arp_ignore

      echo"2">/proc/sys/net/ipv4/conf/lo/arp_announce

      echo"1">/proc/sys/net/ipv4/conf/all/arp_ignore

      echo"2">/proc/sys/net/ipv4/conf/all/arp_announce

      sysctl -p >/dev/null2>&1echo"RealServer Start OK"

      ;;

stop)

      ifconfiglo:0 down

      route del $SNS_VIP >/dev/null2>&1echo"0">/proc/sys/net/ipv4/conf/lo/arp_ignore

      echo"0">/proc/sys/net/ipv4/conf/lo/arp_announce

      echo"0">/proc/sys/net/ipv4/conf/all/arp_ignore

      echo"0">/proc/sys/net/ipv4/conf/all/arp_announce

      echo"RealServer Stoped"      ;;*)

      echo"Usage: $0 {start|stop}"      exit 1esac

exit 0

[root@bogon www]# /etc/rc.d/init.d/realserver.sh start/etc/rc.d/init.d/realserver.sh: line6: /etc/rc.d/init.d/functions: 權限不夠

RealServer Start OK

[root@bogon www]# ifconfigeth0      Link encap:Ethernet  HWaddr 00:0C:29:41:71:DF 

          inet addr:192.168.12.129Bcast:192.168.12.255Mask:255.255.255.0          inet6 addr: fe80::20c:29ff:fe41:71df/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1          RX packets:728errors:0dropped:0overruns:0frame:0          TX packets:98errors:0dropped:0overruns:0carrier:0          collisions:0txqueuelen:1000

          RX bytes:137311(134.0KiB)  TX bytes:7369(7.1 KiB)

eth1      Link encap:Ethernet  HWaddr 00:0C:29:41:71:E9 

          inet addr:192.168.2.204Bcast:192.168.2.255Mask:255.255.255.0          inet6 addr: fe80::20c:29ff:fe41:71e9/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1          RX packets:119838errors:0dropped:0overruns:0frame:0          TX packets:31612errors:0dropped:0overruns:0carrier:0          collisions:0txqueuelen:1000

          RX bytes:23411786(22.3MiB)  TX bytes:2119106(2.0 MiB)

lo        Link encap:Local Loopback 

          inet addr:127.0.0.1Mask:255.0.0.0          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:65536Metric:1          RX packets:2errors:0dropped:0overruns:0frame:0          TX packets:2errors:0dropped:0overruns:0carrier:0          collisions:0txqueuelen:0

          RX bytes:182(182.0b)  TX bytes:182(182.0 b)

lo:0      Link encap:Local Loopback 

          inet addr:192.168.2.13Mask:255.255.255.255          UP LOOPBACK RUNNING  MTU:65536Metric:1

6、啟動keepalived并進行測試

[root@bogon keepalived-1.2.24]# service keepalived start

正在啟動 keepalived：                                      [确定]

lvs-master

[root@bogon keepalived-1.2.24]#tail-f /var/log/messages

Oct 2101:19:46bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:46bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:46bogon Keepalived_healthcheckers[6596]: Netlink reflector reports IP192.168.2.13 added

Oct 2101:19:46bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1(size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port          Forward Weight ActiveConn InActConn

TCP  192.168.2.13:80rr persistent3->192.168.2.204:80Route100->192.168.2.205:80Route100

通路curl http://192.168.2.13/test.txt

[root@www etc]# curl http://192.168.2.13/test.txtit is web2

關掉web2再次測試

[root@www etc]# curl http://192.168.2.13/test.txtit is web1

檢視lvs-master

TCP  192.168.2.13:80rr persistent3->192.168.2.204:80Route102       

Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:28:58bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80failed after1 retry.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80from VS [192.168.2.13]:80Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

Oct 2101:29:31bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.

已經自動把web2剔除

打開web2再次檢視

TCP  192.168.2.13:80rr persistent3->192.168.2.204:80Route100->192.168.2.205:80Route100       

Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:28:58bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 2101:31:01bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 success.

Oct 2101:31:01bogon Keepalived_healthcheckers[6596]: Adding service [192.168.2.205]:80to VS [192.168.2.13]:80Oct 2101:31:01bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25connected.

恢複後已自動添加進來

關掉lvs master的keepalived

[root@bogon keepalived-1.2.24]# service keepalived stop

停止 keepalived：                                          [确定]

通路web并檢視lvs backup

[root@lys2 src]#tail-f /var/log/messages

Oct 2319:03:26lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Transition to MASTER STATE

Oct 2319:03:27lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Entering MASTER STATE

Oct 2319:03:27lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) setting protocol VIPs.

Oct 2319:03:27lys2 Keepalived_healthcheckers[13123]: Netlink reflector reports IP192.168.2.13 added

Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13

[root@lys2 src]# ip addr1: lo: mtu65536 qdisc noqueue state UNKNOWN

    link/loopback00:00:00:00:00:00brd00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever2: eth0: mtu1500qdisc pfifo_fast state UP qlen1000    link/ether00:0c:29:89:0f:e3 brd ff:ff:ff:ff:ff:ff

    inet 192.168.144.101/24brd192.168.144.255 scope global eth0

    inet6 fe80::20c:29ff:fe89:fe3/64 scope link

      valid_lft forever preferred_lft forever3: eth1: mtu1500qdisc pfifo_fast state UP qlen1000    link/ether00:0c:29:89:0f:ed brd ff:ff:ff:ff:ff:ff

    inet 192.168.2.202/24brd192.168.2.255 scope global eth1

    inet 192.168.2.13/32 scope global eth1

    inet6 fe80::20c:29ff:fe89:fed/64 scope link

      valid_lft forever preferred_lft forever

可以看到lvs backup已自動切換成master狀态并自動綁定了vip

檢視lvs master vip

[root@bogon keepalived-1.2.24]# ip addr1: lo: mtu65536 qdisc noqueue state UNKNOWN

      valid_lft forever preferred_lft forever2: eth0: mtu1500qdisc pfifo_fast state UP qlen1000    link/ether00:0c:29:55:4d:7a brd ff:ff:ff:ff:ff:ff

    inet 192.168.12.128/24brd192.168.12.255 scope global eth0

    inet6 fe80::20c:29ff:fe55:4d7a/64 scope link

      valid_lft forever preferred_lft forever3: eth1: mtu1500qdisc pfifo_fast state UP qlen1000    link/ether00:0c:29:55:4d:84 brd ff:ff:ff:ff:ff:ff

    inet 192.168.2.203/24brd192.168.2.255 scope global eth1

    inet6 fe80::20c:29ff:fe55:4d84/64 scope link

已自動解除vip

到處全部結束