天天看點
傳回

Keepalived+LVS(dr)高可用負載均衡叢集的實作一 環境介紹二 原理及拓撲圖三 keepalived配置四 後端RS伺服器的配置五 測試

一 環境介紹

1.作業系統

CentOS Linux release 7.2.1511 (Core)

2.服務

keepalived+lvs雙主高可用負載均衡叢集及LAMP應用

keepalived-1.2.13-7.el7.x86_64

ipvsadm-1.27-7.el7.x86_64

httpd-2.4.6-45.el7.centos.x86_64

mariadb-5.5.52-1.el7.x86_64

php-5.4.16-42.el7.x86_64

二 原理及拓撲圖

1.vrrp協定

vrrp(Virtual Redundant Routing Protocol)協定：

在現實的網絡環境中，兩台需要通信的主機大多數情況下并沒有直接的實體連接配接。對于這樣的情況，它們之間路由怎樣選擇？主機如何標明到達目的主機的下一跳路由，這個問題通常的解決方法有兩種：

 在主機上使用動态路由協定(RIP、OSPF等)

 在主機上配置靜态路由

很明顯，在主機上配置動态路由是非常不切實際的，因為管理、維護成本以及是否支援等諸多問題。配置靜态路由就變得十分流行，但路由器(或者說預設網關default gateway)卻經常成為單點故障。VRRP的目的就是為了解決靜态路由單點故障問題，VRRP通過一競選(election)協定來動态的将路由任務交給LAN中虛拟路由器中的某台VRRP路由器。

2.keepalived簡介

Keepalived 是一個基于VRRP協定來實作的LVS服務高可用方案，可以利用其來避免單點故障。一個LVS服務會有2台伺服器運作Keepalived，一台為主伺服器（MASTER），一台為備份伺服器（BACKUP），但是對外表現為一個虛拟IP，主伺服器會發送特定的消息給備份伺服器，當備份伺服器收不到這個消息的時候，即主伺服器當機的時候，備份伺服器就會接管虛拟IP，繼續提供服務，進而保證了高可用性。Keepalived是VRRP的完美實作。

3.lvs-dr

Direct Routing，直接路由,通過為請求封包重新封裝一個MAC首部進行轉發，源MAC是DIP所在的接口的MAC，目标MAC是某挑選出的RS的RIP所在接口的MAC位址；源IP/PORT，以及目标IP/PORT均保持不變。  

Director和各RS都得配置使用VIP； 

(1) 確定前端路由器将目标IP為VIP的請求封包發往Director；

 (a) 在前端網關做靜态綁定；

 (b) 在RS上使用arptables；

 (c) 在RS上修改核心參數以限制arp通告及應答級别；

  arp_announce

  arp_ignore

(2) RS的RIP可以使用私網位址，也可以是公網位址；RIP與DIP在同一IP網絡；RIP的網關不能指向DIP，以確定響應封包不會經由Director； 

(3) RS跟Director要在同一個實體網絡；

(4) 請求封包要經由Director，但響應不能經由Director，而是由RS直接發往Client；

(5) 不支援端口映射。

4.IP配置設定

VIP1:172.18.67.66

VIP2:172.18.67.88

DIP1:172.18.67.13

DIP2:172.18.67.14

RIP1:172.18.67.11

RIP2:172.18.67.12

CIP:172.18.67.3

5.拓撲圖

Keepalived+LVS(dr)高可用負載均衡叢集的實作一 環境介紹二 原理及拓撲圖三 keepalived配置四 後端RS伺服器的配置五 測試

三 keepalived配置

1.安裝keepalived

[root@inode2 ~]# yum install -y keepalived
[root@inode3 ~]# yum install -y keepalived

2.高可用的ipvs雙主叢集配置  

第一個節點：

[root@inode2 ~]# cd /etc/keepalived/
[root@inode2 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost                  　　　　　　　　　　　　　　#接受通知的郵件位址
}
notification_email_from kaadmin@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node2                 　　　　　　　　　　　　　　 #路由節點
vrrp_mcast_group4 224.0.67.67              　　　　　　　　#多點傳播位址，範圍224.0.0.0~239.255.255.255
}
vrrp_instance myr1 {
state MASTER
interface eno16777736                　　　　　　　　　　　　#網卡接口
virtual_router_id 167                　　　　　　　　　　　　#虛拟路由ID号，0~255
priority 100                  　　　　　　　　　　　　　　　　#優先級，MASTER比BACKUP優先級高
advert_int 1
authentication {
 auth_type PASS
 auth_pass f1bf7fda
}
virtual_ipaddress {
 172.18.67.66/16 dev eno16777736 label eno16777736:0
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"         #調用通知腳本
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance myr2 {
state BACKUP
interface eno16777736
virtual_router_id 168
priority 98
advert_int 1
authentication {
 auth_type PASS
 auth_pass f2bf7ade
}
virtual_ipaddress {
 172.18.67.88/16 dev eno16777736 label eno16777736:1
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 172.18.67.66 80 {              #VIP
delay_loop 2
lb_algo wrr                  　　　　　　　　　　#lvs負載均衡排程算法
lb_kind DR                   　　　　　　　　　　#負載均衡類型
protocol TCP                  　　　　　　　　　 #傳輸協定
sorry_server 127.0.0.1 80               　　　 #localhost
real_server 172.18.67.11 80 {              　  #後端RIP
 weight 1                  　　　　　　　　　　　 #排程權重 
 HTTP_GET {                  　　　　　　　　　　#http請求方式
 url {
  path /
  status_code 200               　　　　　　　　#狀态碼
 }
 connect_timeout 2                　　　　　　  #連接配接逾時
 nb_get_retry 3
 delay_before_retry 3
 }
}
real_server 172.18.67.12 80 {
 weight 1
 HTTP_GET {
  url {
   path /
   status_code 200
  }
  connect_timeout 2
  nb_get_retry 3
  delay_before_retry 3
  }
 }
}

第二個節點：

[root@inode3 ~]# cd /etc/keepalived/
[root@inode3 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from kaadmin@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node3
vrrp_mcast_group4 224.0.67.67
}
vrrp_instance myr1 {
state BACKUP
interface eno16777736
virtual_router_id 167
priority 98
advert_int 1
authentication {
 auth_type PASS
 auth_pass f1bf7fda
}
virtual_ipaddress {
 172.18.67.66/16 dev eno16777736 label eno16777736:0
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance myr2 {
state MASTER
interface eno16777736
virtual_router_id 168
priority 100
advert_int 1
authentication {
 auth_type PASS
 auth_pass f2bf7ade
}
virtual_ipaddress {
 172.18.67.88/16 dev eno16777736 label eno16777736:1
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 172.18.67.88 80 {
delay_loop 2
lb_algo wrr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 172.18.67.11 80 {
 weight 1
 HTTP_GET {
 url {
  path /
  status_code 200
 }
 connect_timeout 2
 nb_get_retry 3
 delay_before_retry 3
 }
}
real_server 172.18.67.12 80 {
 weight 1
 HTTP_GET {
  url {
   path /
   status_code 200
  }
  connect_timeout 2
  nb_get_retry 3
  delay_before_retry 3
  }
 }
}

3.郵件通知腳本

當雙主高可用叢集主備切換時可通過郵件通知管理者，此時在配置檔案中可自動調用實作編輯好的腳本

[root@inode2 ~]# vim notify.sh
#!/bin/bash
#
contact='root@localhost'
notify() {
 mailsubject="$(hostname) to be $1, vip floating"
 mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
 echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
 notify master
 ;;
backup)
 notify backup
 ;;
fault)
 notify fault
 ;;
*)
 echo "Usage: $(basename $0) {master|backup|fault}"
 exit 1
 ;;
esac

節點二同樣配置。

四 後端RS伺服器的配置

1.配置LAMP環境

[root@inode4 ~]# yum install httpd mariadb-server php -y
[root@inode5 ~]# yum install httpd mariadb-server php -y

2.簡單編輯測試網頁

[root@inode4 ~]# echo "RS1:172.18.67.11" > /var/www/html/index.html
[root@inode5 ~]# echo "RS2:172.18.67.12" > /var/www/html/index.html

3.修改RS核心參數

dr模型中，各主機上均需要配置VIP，解決位址沖突的方式有三種：

(1) 在前端網關做靜态綁定；

(2) 在各RS使用arptables；

(3) 在各RS修改核心參數，來限制arp響應和通告的級别；

限制響應級别：arp_ignore

 0：預設值，表示可使用本地任意接口上配置的任意位址進行響應；

 1: 僅在請求的目标IP配置在本地主機的接收到請求封包接口上時，才給予響應；

限制通告級别：arp_announce

 0：預設值，把本機上的所有接口的所有資訊向每個接口上的網絡進行通告；

 1：盡量避免向非直接連接配接網絡進行通告；

 2：必須避免向非本網絡通告。

可通過編輯腳本實作：

[root@inode4 ~]# vim dr.sh
#!/bin/bash
#
case $1 in
start)
 echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
 echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
 echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
 echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
 ;;
stop)
 echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
 echo 0 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
 echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
 echo 0 > /proc/sys/net/ipv4/conf/eth0/arp_announce
 ;;
*) 
 echo "Usage $(basename $0) start|stop"
 exit 1
 ;;
esac

同理第二個RS需同樣配置

4.添加路由

節點一：

[root@inode4 ~]# ifconfig lo:0 172.18.67.66 netmask 255.255.255.255 broadcast 172.18.67.66 up
[root@inode4 ~]# ifconfig lo:1 172.18.67.88 netmask 255.255.255.255 broadcast 172.18.67.88 up
[root@inode4 ~]# route add -host 172.18.67.66 dev lo:0
[root@inode4 ~]# route add -host 172.18.67.88 dev lo:1

節點二：

[root@inode5 ~]# ifconfig lo:0 172.18.67.88 netmask 255.255.255.255 broadcast 172.18.67.88 up
[root@inode5 ~]# ifconfig lo:1 172.18.67.66 netmask 255.255.255.255 broadcast 172.18.67.66 up
[root@inode5 ~]# route add -host 172.18.67.88 dev lo:0
[root@inode5 ~]# route add -host 172.18.67.66 dev lo:1

五 測試

1.啟動服務

RS:

[root@inode4 ~]# systemctl start httpd
[root@inode5 ~]# systemctl start httpd
      
[root@inode2 ~]# systemctl start keepalived.service
[root@inode2 ~]# systemctl status -l  keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-05-14 01:19:27 CST; 17s ago
  Process: 2120 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2121 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─2121 /usr/sbin/keepalived -D
           ├─2122 /usr/sbin/keepalived -D
           └─2123 /usr/sbin/keepalived -D
May 14 01:19:29 inode2 Keepalived_vrrp[2123]: Opening script file /etc/keepalived/notify.sh
May 14 01:19:29 inode2 Keepalived_healthcheckers[2122]: Netlink reflector reports IP 172.18.67.66 added
May 14 01:19:31 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Transition to MASTER STATE
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Entering MASTER STATE
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) setting protocol VIPs.
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: Opening script file /etc/keepalived/notify.sh
May 14 01:19:32 inode2 Keepalived_healthcheckers[2122]: Netlink reflector reports IP 172.18.67.88 added
May 14 01:19:34 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr1) Sending gratuitous ARPs on eno16777736 for 172.18.67.66
May 14 01:19:37 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
[root@inode2 ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:8b:08:6f brd ff:ff:ff:ff:ff:ff
    inet 172.18.67.13/16 brd 172.18.255.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 172.18.67.66/16 scope global secondary eno16777736:0
       valid_lft forever preferred_lft forever
    inet 172.18.67.88/16 scope global secondary eno16777736:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe8b:86f/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
      
[root@inode3 ~]# systemctl start keepalived.service
[root@inode3 ~]# systemctl status -l  keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-05-14 01:20:25 CST; 6s ago
  Process: 2110 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2111 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─2111 /usr/sbin/keepalived -D
           ├─2112 /usr/sbin/keepalived -D
           └─2113 /usr/sbin/keepalived -D
May 14 01:20:25 inode3 Keepalived_vrrp[2113]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Transition to MASTER STATE
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Entering MASTER STATE
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) setting protocol VIPs.
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: Opening script file /etc/keepalived/notify.sh
May 14 01:20:27 inode3 Keepalived_healthcheckers[2112]: Netlink reflector reports IP 172.18.67.88 added
[root@inode3 ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:78:24:c3 brd ff:ff:ff:ff:ff:ff
    inet 172.18.67.14/16 brd 172.18.255.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 172.18.67.88/16 scope global secondary eno16777736:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe78:24c3/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever

2.通路測試

[root@inode1 ~]# curl http://172.18.67.66
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.66
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.66
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.66
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS1:172.18.67.11

高可用負載叢集成功搭建完成。

3.模拟一台web伺服器當機

将RS1的httpd服務停掉，并再此通路：

[root@inode4:~]# systemctl stop httpd
[root@inode1 ~]# curl http://172.18.67.66
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.66
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12

發現照樣可以通路伺服器，實際環境中兩台RS的内容應該是一模一樣的，在這裡我為了以示差別将兩台内容編輯成不同。

4.模拟一台高可用負載叢集當機

将VS1的keepalived服務關閉并測試：

[root@inode2 ~]# systemctl stop keepalived.service
[root@inode1 ~]# curl http://172.18.67.88
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12
[root@inode1 ~]# curl http://172.18.67.88
RS1:172.18.67.11
[root@inode1 ~]# curl http://172.18.67.88
RS2:172.18.67.12

我們發現即使高可用負載均衡叢集中的某一個主機當機了，我們任然可以通過其中的一個IP通路web伺服器，展現出了高可用的實用性，并且在通路中lvs排程器将用戶端請求按設定的權重分别向後端的伺服器實作排程。

負載均衡 網絡協定 排程 開發工具 網絡架構 前端開發 php shell Linux 算法 lvs keepalived搭建叢集 lvs keepalived叢集 lvs keepalived搭建 lvs keepalived搭建通過 lvs keepalived叢集通過
上一篇: Bash腳本程式設計總結
下一篇: 緩存varnish的管理及配置詳解一 工作原理 二 基本架構 三 配置詳解四 通路測試

繼續閱讀