Redis主從自動failover

Redis主從架構持久化存在一個問題，即前次測試的結論，持久化需要配置在主執行個體上才能跨越執行個體保證資料不丢失，這樣以來主執行個體在持久化資料到硬碟的過程中，勢必會造成磁盤的I/O等待，經過實際測試，這個持久化寫硬碟的過程給應用程式帶來的影響無法忍受；因而在大多數場景下，會考慮把持久化配置在從執行個體上，當主執行個體當機後，通過手動或者自動的方式将從執行個體提升為主執行個體，繼續提供服務！當主執行個體恢複後，先從原從執行個體上同步資料，同步完成後再恢複到原始的主從狀态！要實作這種的要求，需要有keepalive的配合，一方面keepalive提供了VIP，可以避免修改應用程式連接配接，同時redis執行個體的配置檔案監聽部分也需要修改為全網監聽；另一方面keepalive定時排程腳本來監控主從執行個體的狀态，根據具體情況進行切換！本文将重點介紹下使用keepalive實作redis主從自動failover！

環境介紹

作業系統版本均為：rhel5.4 64bit

redis版本：2.6.4

redis執行個體端口均為：6379

redis執行個體密碼均為：123

VIP：192.168.1.120

主執行個體為server11(192.168.1.112)

從執行個體為server12(192.168.1.113，開啟快照持久化)

一：安裝keepalive軟體,server11安裝完成後直接scp至server12上即可

[root@server11 ~]# wget http://keepalived.org/software/keepalived-1.1.19.tar.gz

[root@server11 ~]# tar -zxvf ../tarbag/keepalived-1.1.19.tar.gz

[root@server11 ~]# cd keepalived-1.1.19/

[root@server11 ~]# ./configure --prefix=/usr/local/keepalived && make && make install

二：配置主節點server11配置檔案

[root@server11 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

router_id LVS_DEVEL

}

vrrp_script Monitor_redis {

script "/usr/local/scripts/redis_monitor.sh"

interval 2

weight 2

vrrp_instance VI_1{

state MASTER

interface eth0

virtual_router_id 51

mcast_src_ip 192.168.1.112

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass password_123

track_script {

Monitor_redis

virtual_ipaddress {

192.168.1.120

}

notify_fault /usr/local/scripts/redis_fault.sh

notify_stop /usr/local/scripts/redis_stop.sh

}

三：配置從節點server12配置檔案

[root@server12 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf

state BACKUP

mcast_src_ip 192.168.1.113

priority 99

notify_master /usr/local/scripts/redis_master.sh

notify_backup /usr/local/scripts/redis_backup.sh

四：準備相關的腳本，主從執行個體上都需要存在這些腳本，同時注意腳本需要由可執行權限

[root@server11 ~]# cat /usr/local/scripts/redis_monitor.sh

#!/bin/bash

ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123 PING)

if [ "$ALIVE" == "PONG" ]; then

echo $ALIVE

exit 0

else

killall -9 keepalived

service network restart

exit 1

[root@server11 ~]# sh /usr/local/scripts/redis_monitor.sh

PONG

[root@server11 ~]# cat /usr/local/scripts/redis_master.sh

REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123"

LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"

echo "[master]" >> $LOGFILE

date >> $LOGFILE

echo "Being master...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE 2>&1

sleep 10

echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

[root@server11 ~]# cat /usr/local/scripts/redis_backup.sh

echo "[backup]" >> $LOGFILE

echo "Being slave...." >> $LOGFILE 2>&1

sleep 15

$REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE 2>&1

[root@server11 ~]# cat /usr/local/scripts/redis_stop.sh

#!/bin/bash

echo "[stop]" >> $LOGFILE

date >> $LOGFILE

[root@server11 ~]# cat /usr/local/scripts/redis_fault.sh

#!/bin/bash

LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"

echo "[fault]" >> $LOGFILE

date >> $LOGFILE

五：主從執行個體分别啟動keepalive程序，測試VIP是否正常(這裡就要修改redis配置檔案的監聽位址為0.0.0.0)

[root@server11 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf

[root@server11 ~]# tail -f /var/log/messages

Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Configuration is using : 5499 Bytes

Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Using LinkWatch kernel netlink reflector...

Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]

Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP_Script(Monitor_redis) succeeded

Dec 12 09:25:50 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Transition to MASTER STATE

Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Entering MASTER STATE

Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) setting protocol VIPs.

Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120

Dec 12 09:25:51 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.

Dec 12 09:25:51 server11 Keepalived_healthcheckers[7710]: Netlink reflector reports IP 192.168.1.120 added

Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: Netlink reflector reports IP 192.168.1.120 added

Dec 12 09:25:56 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120

[root@server11 ~]# ip a |grep 192

inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0

inet 192.168.1.120/32 scope global eth0

[root@server12 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf

[root@server12 ~]# tail -f /var/log/messages

Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Configuration is using : 5595 Bytes

Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE

Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Using LinkWatch kernel netlink reflector...

Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]

Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Script(Monitor_redis) succeeded

[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'

# Replication

role:master

connected_slaves:1

slave0:192.168.1.113,6379,online

六：主執行個體寫入測試資料，該腳本原則上會寫入25條測試資料，不過由于未優化redis預設并發數，會導緻一些寫入請求失敗，最終功寫入231839條測試資料，占記憶體總大小為25M左右，寫入過程中可以觀察主從執行個體的持久化檔案變化情況，主執行個體的持久化檔案維持在30k，從執行個體的則不斷的擴充！

[root@server11 ~]# cat test.sh

REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 -n 1 SET"

ID=1

while(($ID<50001))

INSTANCE_NAME="i-2-$ID-VM"

UUID=`cat /proc/sys/kernel/random/uuid`

PRIVATE_IP_ADDRESS=10.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`.`echo

"$RANDOM % 255 + 1" | bc`\

CREATED=`date "+%Y-%m-%d %H:%M:%S"`

$REDISCLI vm_instance:$ID:instance_name "$INSTANCE_NAME"

$REDISCLI vm_instance:$ID:uuid "$UUID"

$REDISCLI vm_instance:$ID:private_ip_address "$PRIVATE_IP_ADDRESS"

$REDISCLI vm_instance:$ID:created "$CREATED"

$REDISCLI vm_instance:$INSTANCE_NAME:id "$ID"

ID=$(($ID+1))

done

[root@server11 ~]# sh test.sh

[root@server11 redis2]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep

'used_memory_peak_human|db1:keys'

used_memory_peak_human:24.98M

db1:keys=231839,expires=0

七：模拟主執行個體故障，觀察日志輸出，驗證從執行個體是否能成功接管VIP，同時将執行個體變成讀寫模式

[root@server11 ~]# killall -9 redis-server

[root@server11 ~]# ps -ef |grep redis

root 15886 6458 0 09:49 pts/0 00:00:00 grep redis

[root@server11 ~]# ps -ef |grep keep

root 16029 6458 0 09:49 pts/0 00:00:00 grep keep

[root@server12 ~]# tail -f /usr/local/redis2/var/keepalived-redis-state.log

[master]

Wed Dec 12 09:48:52 CST 2012

Being master....

Run SLAVEOF cmd ...

OK Already connected to specified master

Run SLAVEOF NO ONE cmd ...

Dec 12 09:48:51 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Transition to MASTER STATE

Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering MASTER STATE

Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) setting protocol VIPs.

Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120

Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: Netlink reflector reports IP 192.168.1.120 added

Dec 12 09:48:52 server12 avahi-daemon[2921]: Registering new address record for 192.168.1.120 on eth0.

Dec 12 09:48:52 server12 Keepalived_healthcheckers[3106]: Netlink reflector reports IP 192.168.1.120 added

Dec 12 09:48:57 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120

[root@server12 ~]# ip a |grep 192

inet 192.168.1.113/24 brd 192.168.1.255 scope global eth0

[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'

connected_slaves:0

[root@server12 ~]# sh test.sh

[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep

used_memory_peak_human:26.78M

db1:keys=249925,expires=0

九:主執行個體角色的恢複過程，使用shell腳本自動恢複

[root@server11 ~]# ssh-keygen

[root@server11 ~]# cd .ssh/

[root@server11 .ssh]# ssh-copy-id -i id_rsa.pub [email protected]

[root@server11 ~]# cat /usr/local/scripts/recover_mastart.sh

#!/bin/sh

ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.113 -p 6379 -a 123 PING)

MDB=/usr/local/redis2/master_dump.rdb

SDB=/usr/local/redis2/slave_dump.rdb

scp [email protected]:$SDB $MDB

echo $ALIVE

/usr/local/redis2/bin/redis-server /usr/local/redis2/etc/redis.conf

/usr/local/keepalived/sbin/keepalived -D -f

/usr/local/keepalived/etc/keepalived/keepalived.conf

[root@server11 ~]# chmod +x /usr/local/scripts/recover_mastart.sh

[root@server11 ~]# sh /usr/local/scripts/recover_mastart.sh

十：驗證資料完整性和主從角色恢複情況

[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep 'used_memory_peak_human|db1:keys'

[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'

slave0:192.168.1.113,6379,online

[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 3 'Replication'

role:slave

master_host:192.168.1.112

master_port:6379

[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep 'used_memory_peak_human|db1:keys'

主執行個體keepalive日志：

Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]

Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Script(Monitor_redis) succeeded

Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Transition to MASTER STATE

Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Received higher prio advert

Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Entering BACKUP STATE

Dec 12 10:08:15 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) forcing a new MASTER election

Dec 12 10:08:16 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Transition to MASTER STATE

Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Entering MASTER STATE

Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) setting protocol VIPs.

Dec 12 10:08:17 server11 Keepalived_healthcheckers[20230]: Netlink reflector reports IP 192.168.1.120 added

Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120

Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: Netlink reflector reports IP 192.168.1.120 added

Dec 12 10:08:17 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.

從執行個體keepalive日志：

Dec 12 09:56:01 server12 last message repeated 4 times

Dec 12 10:08:13 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Received lower prio advert, forcing new election

Dec 12 10:08:13 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120

Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Received higher prio advert

Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE

Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) removing protocol VIPs.

Dec 12 10:08:15 server12 Keepalived_healthcheckers[3106]: Netlink reflector reports IP 192.168.1.120 removed

Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: Netlink reflector reports IP 192.168.1.120 removed

Dec 12 10:08:15 server12 avahi-daemon[2921]: Withdrawing address record for 192.168.1.120 on eth0.

從執行個體角色轉換日志：

[backup]

Wed Dec 12 10:08:15 CST 2012

Being slave....

參考文檔：

<a href="http://heylinux.com/archives/1932.html">http://heylinux.com/archives/1932.html</a>

本文轉自斬月部落格51CTO部落格，原文連結http://blog.51cto.com/ylw6006/1086455如需轉載請自行聯系原作者

ylw6006

Redis主從自動failover

繼續閱讀

資料遷移方法資料遷移原則資料遷移之雙寫方案資料遷移之級聯同步方案

微服務-性能壓測\緩存redis和分布式鎖redisson和SpringCache

Nacos 2.0 更新前後性能對比壓測

Spring資料和Redis

redis叢集資料一緻性_RedisRaft為Redis叢集帶來強大的資料一緻性

HK-2000資料采集儀資料庫操作說明

終端環境之tmux

supervisor 管理redis 和httpd 環境centos7

Redis簡介一(單機版)發展曆程Redis

查找檔案中的字元串

拒絕使用者登入:/bin/false和/usr/sbin/nologin

Shell程式設計——sort排序、uniq忽略重複、tr替換壓縮删除、cut指定删除字段、正規表達式元字元sort 指令uniq 指令tr 指令cut 指令正規表達式

Linxu常用指令技巧彙總

《Linux指令行與Shell腳本程式設計大全第2版.布盧姆》pdf

ACS基本配置-權限等級管理

Bugku-WEB-web33