天天看點
傳回

實作redis哨兵，模拟master故障場景1、redis哨兵（Sentinel）一鍵安裝redis的腳本

1、redis哨兵（Sentinel）

實作redis哨兵，模拟master故障場景1、redis哨兵（Sentinel）一鍵安裝redis的腳本

1.1、redis叢集介紹

前面文章講的主從複制叢集是無法實作master和slave角色的自動切換的，如果master節點出現現redis服務異常、主機斷電、磁盤損壞等問題導緻master無法使用，而redis主從複制無法實作自動的故障轉移(将slave 自動提升為新master)，需要手動修改環境配置,才能切換到slave redis伺服器，另外當單台Redis伺服器性能無法滿足業務寫入需求的時候,也無法橫向擴充Redis服務的并行寫入性能。

需要解決以上的兩個核心問題：

  • master和slave角色的無縫切換，讓業務無感覺進而不影響業務使用；
  • 可橫向動态擴充Redis伺服器，進而實作多台伺服器并行寫入以實作更高并發的目的。

Redis叢集實作的方式：

  • 用戶端分片: 由應用決定将不同的KEY發送到不同的Redis伺服器
  • 代理分片: 由代理決定将不同的KEY發送到不同的Redis伺服器,代理程式如:codis,twemproxy等
  • Redis Cluster

1.2、redis哨兵（Sentinel）的工作原理

Sentinel可以管理多個redis主從叢集

實作redis哨兵，模拟master故障場景1、redis哨兵（Sentinel）一鍵安裝redis的腳本

Sentinel 架構

實作redis哨兵，模拟master故障場景1、redis哨兵（Sentinel）一鍵安裝redis的腳本

Sentinel 故障轉移

實作redis哨兵，模拟master故障場景1、redis哨兵（Sentinel）一鍵安裝redis的腳本

Sentinel 程序是用于監控redis叢集中Master主伺服器工作的狀态，在Master主伺服器發生故障的時候，可以實作Master和Slave伺服器的切換，保證系統的高可用，此功能在redis2.6+的版本已引用，Redis的哨兵模式到了2.8版本之後就穩定了下來。一般在生産環境也建議使用Redis的2.8版本的以後版本

哨兵(Sentinel) 是一個分布式系統，可以在一個架構中運作多個哨兵(sentinel) 程序，這些程序使用流言協定(gossip protocols)來接收關于Master主伺服器是否下線的資訊，并使用投票協定(Agreement Protocols)來決定是否執行自動故障遷移,以及選擇哪個Slave作為新的Master

每個哨兵(Sentinel)程序會向其它哨兵(Sentinel)、Master、Slave定時發送消息，以确認對方是否”活”着，如果發現對方在指定配置時間(此項可配置)内未得到回應，則暫時認為對方已離線，也就是所謂的”主觀認為當機” (主觀:是每個成員都具有的獨自的而且可能相同也可能不同的意識)，英文名稱：Subjective Down，簡稱SDOWN

有主觀當機，對應的有客觀當機。當“哨兵群”中的多數Sentinel程序在對Master主伺服器做出SDOWN 的判斷，并且通過 SENTINEL is-master-down-by-addr 指令互相交流之後，得出的Master Server下線判斷，這種方式就是“客觀當機”(客觀:是不依賴于某種意識而已經實際存在的一切事物)，英文名稱是：Objectively Down， 簡稱 ODOWN

通過一定的vote算法，從剩下的slave從伺服器節點中，選一台提升為Master伺服器節點，然後自動修改相關配置，并開啟故障轉移（failover）

Sentinel 機制可以解決master和slave角色的自動切換問題，但單個 Master 的性能瓶頸問題無法解決,類似于MySQL中的MHA功能

Redis Sentinel中的Sentinel節點個數應該為大于等于3且最好為奇數

用戶端初始化時連接配接的是Sentinel節點集合，不再是具體的Redis節點，但Sentinel隻是配置中心不是代理。

Redis Sentinel節點與普通redis沒有差別,要實作讀寫分離依賴于用戶端程式

redis 3.0之前版本中,生産環境一般使用哨兵模式,3.0後推出redis cluster功能,可以支援更大規模的生産環境

sentinel中的三個定時任務：

  • 每10秒每個sentinel對master和slave執行info

    發現slave節點

    确認主從關系

  • 每2秒每個sentinel通過master節點的channel交換資訊(pub/sub)

    通過sentinel__:hello頻道互動

    互動對節點的“看法”和自身資訊

  • 每1秒每個sentinel對其他sentinel和redis執行ping

1.3、實作哨兵

實作redis哨兵，模拟master故障場景1、redis哨兵（Sentinel）一鍵安裝redis的腳本

環境準備：

準備三台主機搭建主從叢集，再在每個機器上搭建sentinel

IP Redis版本 主機名
10.0.0.100 redis-6.2.7 node1.stars.org
10.0.0.101 redis-6.2.7 node2.stars.org
10.0.0.102 redis-6.2.7 node3.stars.org

1.3.1、實作哨兵需要先實作一下主從複制的架構

哨兵的前提是已經實作了一個redis的主從複制的運作環境，進而實作一個一主兩從基于哨兵的高可用redis架構

注意: master的配置檔案中masterauth和slave都必須相同

所有主從節點的redis.conf中關健配置如下：

#在所有主從節點上執行
root@node1:~# vim /apps/redis/etc/redis.conf
bind 0.0.0.0
masterauth wm521314
requirepass wm521314
#這裡也可以使用sed修改
#sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e 's/^# masterauth .*/masterauth 123456/' -e 's/^# requirepass .*/requirepass 123456/' /apps/redis/etc/redis.conf

#在所有的從節點執行
root@node2:~# echo "replicaof 10.0.0.100 6379" >> /apps/redis/etc/redis.conf

#在所有主從節點上執行
root@node1:~# systemctl restart redis.service

檢視master伺服器狀态：

root@node1:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:master
connected_slaves:2
slave0:ip=10.0.0.101,port=6379,state=online,offset=126,lag=0
slave1:ip=10.0.0.102,port=6379,state=online,offset=126,lag=0
master_failover_state:no-failover
master_replid:6c38e5bab8f2ebd438426c7c46556eae889b46ea
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:126
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:126

檢視slave1伺服器狀态：

root@node2:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:10.0.0.100
master_port:6379
master_link_status:up
master_last_io_seconds_ago:9
master_sync_in_progress:0
slave_read_repl_offset:238
slave_repl_offset:238
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:6c38e5bab8f2ebd438426c7c46556eae889b46ea
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:238
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:238

檢視slave1伺服器狀态：

root@node3:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:10.0.0.100
master_port:6379
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_read_repl_offset:280
slave_repl_offset:280
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:6c38e5bab8f2ebd438426c7c46556eae889b46ea
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:280
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:113
repl_backlog_histlen:168

1.3.2、編輯哨兵的配置檔案

Sentinel實際上是一個特殊的redis伺服器,有些redis指令支援,但很多指令并不支援.預設監聽在26379/tcp端口.

所有redis節點使用相同的以下的配置檔案

#這裡因為我是源碼編譯安裝的，是以要人源碼的路徑下找到sentinel.conf将其複制到redis安裝的目錄下
root@node1:~# cd /usr/local/src/redis-6.2.7/
root@node1:/usr/local/src/redis-6.2.7# ls
00-RELEASENOTES  CONTRIBUTING  INSTALL    README.md   runtest-cluster    sentinel.conf  TLS.md
BUGS             COPYING       Makefile   redis.conf  runtest-moduleapi  src            utils
CONDUCT          deps          MANIFESTO  runtest     runtest-sentinel   tests
root@node1:/usr/local/src/redis-6.2.7# cp sentinel.conf /apps/redis/etc/redis-sentinel.conf
root@node1:/usr/local/src/redis-6.2.7# cd
root@node1:~# vim /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile /apps/redis/run/redis-sentinel.pid
logfile /apps/redis/log/sentinel.log
dir /tmp
sentinel monitor mymaster 10.0.0.100 6379 2	#mymaster是叢集的名稱，此行指定目前mymaster叢集中master伺服器的位址和端口
#2為法定人數限制(quorum)，即有幾個sentinel認為master down了就進行故障轉移，一般此值是所有sentinel節點(一般總數是>=3的 奇數,如:3,5,7等)的一半以上的整數值，比如，總數是3，即3/2=1.5，取整為2,是master的ODOWN客觀下線的依據
sentinel auth-pass mymaster wm521314	#mymaster叢集中master的密碼，注意此行要在上面行的下面
sentinel down-after-milliseconds mymaster 3000	#(SDOWN)判斷mymaster叢集中所有節點的主觀下線的時間，機關：毫秒，建議3000
sentinel parallel-syncs mymaster 1	#發生故障轉移後，可以同時向新master同步資料的slave的數量，數字越小總同步時間越長，但可以減輕新master的負載壓力
sentinel failover-timeout mymaster 180000	#所有slaves指向新的master所需的逾時時間，機關：毫秒
sentinel deny-scripts-reconfig yes	#禁止修改腳本

root@node1:~# scp /apps/redis/etc/redis-sentinel.conf 10.0.0.101:/apps/redis/etc/
root@node1:~# scp /apps/redis/etc/redis-sentinel.conf 10.0.0.102:/apps/redis/etc/

三個哨兵伺服器的配置都如下：

root@node1:~# egrep -v "^#|^$" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile /apps/redis/run/redis-sentinel.pid
logfile /apps/redis/log/sentinel.log
dir /tmp
sentinel monitor mymaster 10.0.0.100 6379 2
sentinel auth-pass mymaster wm521314
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no

1.3.2、啟動哨兵

三台哨兵伺服器都要啟動

#我這裡是編譯安裝，是以在所有節點生成新的service檔案
root@node1:~# vim /lib/systemd/system/redis-sentinel.service
[Unit]
Description=Redis Sentinel
After=network.target

[Service]
ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/redis-sentinel.conf --supervised systemd
ExecStop=/bin/kill -s QUIT $MAINPID
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

root@node1:~# scp /lib/systemd/system/redis-sentinel.service 10.0.0.101:/lib/systemd/system/
root@node1:~# scp /lib/systemd/system/redis-sentinel.service 10.0.0.102:/lib/systemd/system/

#剛剛複制了配置檔案到redis安裝目錄了，這裡需要把目錄的權限修改一下，不然服務啟動不了，下面修改權限是所有的節點都要修改
root@node1:~# chown -R redis.redis /apps/redis/
root@node2:~# chown -R redis.redis /apps/redis/
root@node3:~# chown -R redis.redis /apps/redis/

#在所有的主從節點都重新加載以下配置檔案，并設定服務開機自啟動
root@node1:~# systemctl daemon-reload
root@node1:~# systemctl enable --now redis-sentinel.service

#如果是編譯安裝，在所有哨兵伺服器執行下面操作也可以啟動哨兵
root@node1:~# /apps/redis/bin/redis-sentinel /apps/redis/etc/redis-sentinel.conf

1.3.3、驗證哨兵端口 

root@node1:~# ss -tln
State       Recv-Q       Send-Q              Local Address:Port              Peer Address:Port       
LISTEN      0            128                       0.0.0.0:42804                  0.0.0.0:*          
LISTEN      0            128                 127.0.0.53%lo:53                     0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:22                     0.0.0.0:*          
LISTEN      0            128                     127.0.0.1:6010                   0.0.0.0:*          
LISTEN      0            64                        0.0.0.0:36864                  0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:24576                  0.0.0.0:*          
LISTEN      0            64                        0.0.0.0:2049                   0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:52456                  0.0.0.0:*          
LISTEN      0            511                       0.0.0.0:26379                  0.0.0.0:*          
LISTEN      0            511                       0.0.0.0:6379                   0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:111                    0.0.0.0:*          
LISTEN      0            128                          [::]:59698                     [::]:*          
LISTEN      0            128                          [::]:22                        [::]:*          
LISTEN      0            128                         [::1]:6010                      [::]:*          
LISTEN      0            128                          [::]:22746                     [::]:*          
LISTEN      0            64                           [::]:2049                      [::]:*          
LISTEN      0            64                           [::]:35426                     [::]:*          
LISTEN      0            128                          [::]:58824                     [::]:*          
LISTEN      0            128                          [::]:111                       [::]:*

root@node2:~# ss -tln
State       Recv-Q       Send-Q              Local Address:Port              Peer Address:Port       
LISTEN      0            511                       0.0.0.0:26379                  0.0.0.0:*          
LISTEN      0            511                       0.0.0.0:6379                   0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:111                    0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:19474                  0.0.0.0:*          
LISTEN      0            64                        0.0.0.0:28020                  0.0.0.0:*          
LISTEN      0            128                 127.0.0.53%lo:53                     0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:22                     0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:29496                  0.0.0.0:*          
LISTEN      0            128                     127.0.0.1:6010                   0.0.0.0:*          
LISTEN      0            64                        0.0.0.0:2049                   0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:53602                  0.0.0.0:*          
LISTEN      0            128                          [::]:57036                     [::]:*          
LISTEN      0            128                          [::]:111                       [::]:*          
LISTEN      0            128                          [::]:53136                     [::]:*          
LISTEN      0            128                          [::]:22                        [::]:*          
LISTEN      0            128                         [::1]:6010                      [::]:*          
LISTEN      0            128                          [::]:11002                     [::]:*          
LISTEN      0            64                           [::]:21372                     [::]:*          
LISTEN      0            64                           [::]:2049                      [::]:*

root@node3:~# ss -tln
State       Recv-Q       Send-Q              Local Address:Port              Peer Address:Port       
LISTEN      0            128                       0.0.0.0:57072                  0.0.0.0:*          
LISTEN      0            128                 127.0.0.53%lo:53                     0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:22                     0.0.0.0:*          
LISTEN      0            128                     127.0.0.1:6010                   0.0.0.0:*          
LISTEN      0            64                        0.0.0.0:2049                   0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:60290                  0.0.0.0:*          
LISTEN      0            64                        0.0.0.0:14852                  0.0.0.0:*          
LISTEN      0            511                       0.0.0.0:26379                  0.0.0.0:*          
LISTEN      0            511                       0.0.0.0:6379                   0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:15822                  0.0.0.0:*          
LISTEN      0            128                       0.0.0.0:111                    0.0.0.0:*          
LISTEN      0            128                          [::]:22                        [::]:*          
LISTEN      0            128                         [::1]:6010                      [::]:*          
LISTEN      0            128                          [::]:32992                     [::]:*          
LISTEN      0            128                          [::]:32096                     [::]:*          
LISTEN      0            64                           [::]:2049                      [::]:*          
LISTEN      0            128                          [::]:42850                     [::]:*          
LISTEN      0            64                           [::]:20716                     [::]:*          
LISTEN      0            128                          [::]:111                       [::]:*

1.3.4、檢視哨兵日志 

root@node1:~# tail -f /apps/redis/log/sentinel.log 
1298:X 23 May 2022 22:24:23.240 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1298:X 23 May 2022 22:24:23.241 * monotonic clock: POSIX clock_gettime
1298:X 23 May 2022 22:24:23.241 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
1298:X 23 May 2022 22:24:23.241 * Running mode=sentinel, port=26379.
1298:X 23 May 2022 22:24:23.251 # Sentinel ID is 919e8fe5e9963e805552e879847c3274ed8a040c
1298:X 23 May 2022 22:24:23.251 # +monitor master mymaster 10.0.0.100 6379 quorum 2
1298:X 23 May 2022 22:24:23.252 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:24:23.253 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:24:25.253 * +sentinel sentinel c8d2fb34edbf68e35022c915563a4f9693d19410 10.0.0.101 26379 @ mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:24:25.300 * +sentinel sentinel 8a281d471b4d21f123dae1edf58cab8585b95e9c 10.0.0.102 26379 @ mymaster 10.0.0.100 6379
           
root@node2:~# tail -f /apps/redis/log/sentinel.log 
3478:X 23 May 2022 14:24:23.248 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
3478:X 23 May 2022 14:24:23.248 * monotonic clock: POSIX clock_gettime
3478:X 23 May 2022 14:24:23.248 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
3478:X 23 May 2022 14:24:23.248 * Running mode=sentinel, port=26379.
3478:X 23 May 2022 14:24:23.260 # Sentinel ID is c8d2fb34edbf68e35022c915563a4f9693d19410
3478:X 23 May 2022 14:24:23.260 # +monitor master mymaster 10.0.0.100 6379 quorum 2
3478:X 23 May 2022 14:24:23.260 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
3478:X 23 May 2022 14:24:23.261 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.100 6379
3478:X 23 May 2022 14:24:25.263 * +sentinel sentinel 919e8fe5e9963e805552e879847c3274ed8a040c 10.0.0.100 26379 @ mymaster 10.0.0.100 6379
3478:X 23 May 2022 14:24:25.295 * +sentinel sentinel 8a281d471b4d21f123dae1edf58cab8585b95e9c 10.0.0.102 26379 @ mymaster 10.0.0.100 6379
           
root@node3:~# tail -f /apps/redis/log/sentinel.log 
1466:X 23 May 2022 22:24:23.251 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1466:X 23 May 2022 22:24:23.251 * monotonic clock: POSIX clock_gettime
1466:X 23 May 2022 22:24:23.251 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
1466:X 23 May 2022 22:24:23.251 * Running mode=sentinel, port=26379.
1466:X 23 May 2022 22:24:23.261 # Sentinel ID is 8a281d471b4d21f123dae1edf58cab8585b95e9c
1466:X 23 May 2022 22:24:23.261 # +monitor master mymaster 10.0.0.100 6379 quorum 2
1466:X 23 May 2022 22:24:23.262 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:24:23.262 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:24:25.285 * +sentinel sentinel c8d2fb34edbf68e35022c915563a4f9693d19410 10.0.0.101 26379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:24:25.300 * +sentinel sentinel 919e8fe5e9963e805552e879847c3274ed8a040c 10.0.0.100 26379 @ mymaster 10.0.0.100 6379

1.3.5、目前sentinel狀态 

root@node1:~# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.100:6379,slaves=2,sentinels=3

1.3.6、驗證資料是否同步 

root@node1:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> set name ZG
OK
127.0.0.1:6379> set age 22
OK

root@node2:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> KEYS *
1) "age"
2) "name"
127.0.0.1:6379> get name
"ZG"
127.0.0.1:6379> get age
"22"

root@node3:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> KEYS *
1) "age"
2) "name"
127.0.0.1:6379> get name
"ZG"
127.0.0.1:6379> get age
"22"

1.3.7、制造Redis Master節點故障故障轉移并檢視其過程 

root@node1:~# killall redis-server

#觀察日志，這裡會看到10.0.0.102變成新的master了
root@node1:~# tail -f /apps/redis/log/sentinel.log
1298:X 23 May 2022 22:24:23.240 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1298:X 23 May 2022 22:24:23.241 * monotonic clock: POSIX clock_gettime
1298:X 23 May 2022 22:24:23.241 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
1298:X 23 May 2022 22:24:23.241 * Running mode=sentinel, port=26379.
1298:X 23 May 2022 22:24:23.251 # Sentinel ID is 919e8fe5e9963e805552e879847c3274ed8a040c
1298:X 23 May 2022 22:24:23.251 # +monitor master mymaster 10.0.0.100 6379 quorum 2
1298:X 23 May 2022 22:24:23.252 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:24:23.253 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:24:25.253 * +sentinel sentinel c8d2fb34edbf68e35022c915563a4f9693d19410 10.0.0.101 26379 @ mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:24:25.300 * +sentinel sentinel 8a281d471b4d21f123dae1edf58cab8585b95e9c 10.0.0.102 26379 @ mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:39:35.874 # +sdown master mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:39:35.876 # +new-epoch 1
1298:X 23 May 2022 22:39:35.878 # +vote-for-leader 8a281d471b4d21f123dae1edf58cab8585b95e9c 1
1298:X 23 May 2022 22:39:35.952 # +odown master mymaster 10.0.0.100 6379 #quorum 3/2
1298:X 23 May 2022 22:39:35.952 # Next failover delay: I will not start a failover before Mon May 23 22:45:36 2022
1298:X 23 May 2022 22:39:36.977 # +config-update-from sentinel 8a281d471b4d21f123dae1edf58cab8585b95e9c 10.0.0.102 26379 @ mymaster 10.0.0.100 6379
1298:X 23 May 2022 22:39:36.977 # +switch-master mymaster 10.0.0.100 6379 10.0.0.102 6379
1298:X 23 May 2022 22:39:36.977 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
1298:X 23 May 2022 22:39:36.977 * +slave slave 10.0.0.100:6379 10.0.0.100 6379 @ mymaster 10.0.0.102 6379
1298:X 23 May 2022 22:39:40.038 # +sdown slave 10.0.0.100:6379 10.0.0.100 6379 @ mymaster 10.0.0.102 6379

#檢視各節點上哨兵資訊：
root@node1:~# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.102:6379,slaves=2,sentinels=3

#故障轉移後的redis配置檔案會被自動修改
root@node2:~# grep ^replicaof /apps/redis/etc/redis.conf
replicaof 10.0.0.102 6379

#哨兵配置檔案的sentinel monitor IP 同樣也會被修改
root@node2:~# grep "^[a-Z]" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel monitor mymaster 10.0.0.102 6379 2
sentinel auth-pass mymaster wm521314
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
maxclients 4064
user default on nopass ~* &* +@all
sentinel myid c8d2fb34edbf68e35022c915563a4f9693d19410
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1
sentinel known-replica mymaster 10.0.0.100 6379
sentinel known-replica mymaster 10.0.0.101 6379
sentinel known-sentinel mymaster 10.0.0.102 26379 8a281d471b4d21f123dae1edf58cab8585b95e9c
sentinel known-sentinel mymaster 10.0.0.100 26379 919e8fe5e9963e805552e879847c3274ed8a040c

#進入redis檢視現在info replication資訊
root@node3:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.101,port=6379,state=online,offset=345542,lag=1
master_failover_state:no-failover
master_replid:9e7e93af69bee27494f2592f2f45ae5475583f8a
master_replid2:9935b9865c7268c1aac772d2ed2a0deea625cdb5
master_repl_offset:345542
second_repl_offset:182386
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:345542

root@node2:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:10.0.0.102
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:331269
slave_repl_offset:331269
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:9e7e93af69bee27494f2592f2f45ae5475583f8a
master_replid2:9935b9865c7268c1aac772d2ed2a0deea625cdb5
master_repl_offset:331269
second_repl_offset:182386
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:331269

1.3.8、檢視選舉新主的過程日志監測 

root@node3:~# tail -f /apps/redis/log/sentinel.log
1466:X 23 May 2022 22:24:23.251 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1466:X 23 May 2022 22:24:23.251 * monotonic clock: POSIX clock_gettime
1466:X 23 May 2022 22:24:23.251 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
1466:X 23 May 2022 22:24:23.251 * Running mode=sentinel, port=26379.
1466:X 23 May 2022 22:24:23.261 # Sentinel ID is 8a281d471b4d21f123dae1edf58cab8585b95e9c
1466:X 23 May 2022 22:24:23.261 # +monitor master mymaster 10.0.0.100 6379 quorum 2
1466:X 23 May 2022 22:24:23.262 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:24:23.262 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:24:25.285 * +sentinel sentinel c8d2fb34edbf68e35022c915563a4f9693d19410 10.0.0.101 26379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:24:25.300 * +sentinel sentinel 919e8fe5e9963e805552e879847c3274ed8a040c 10.0.0.100 26379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:35.789 # +sdown master mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:35.866 # +odown master mymaster 10.0.0.100 6379 #quorum 2/2
1466:X 23 May 2022 22:39:35.866 # +new-epoch 1
1466:X 23 May 2022 22:39:35.866 # +try-failover master mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:35.869 # +vote-for-leader 8a281d471b4d21f123dae1edf58cab8585b95e9c 1
1466:X 23 May 2022 22:39:35.874 # 919e8fe5e9963e805552e879847c3274ed8a040c voted for 8a281d471b4d21f123dae1edf58cab8585b95e9c 1
1466:X 23 May 2022 22:39:35.874 # c8d2fb34edbf68e35022c915563a4f9693d19410 voted for 8a281d471b4d21f123dae1edf58cab8585b95e9c 1
1466:X 23 May 2022 22:39:35.925 # +elected-leader master mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:35.925 # +failover-state-select-slave master mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:35.992 # +selected-slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:35.992 * +failover-state-send-slaveof-noone slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:36.077 * +failover-state-wait-promotion slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:36.889 # +promoted-slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:36.889 # +failover-state-reconf-slaves master mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:36.971 * +slave-reconf-sent slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:37.898 * +slave-reconf-inprog slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:37.898 * +slave-reconf-done slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:37.973 # -odown master mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:37.973 # +failover-end master mymaster 10.0.0.100 6379
1466:X 23 May 2022 22:39:37.973 # +switch-master mymaster 10.0.0.100 6379 10.0.0.102 6379
1466:X 23 May 2022 22:39:37.973 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
1466:X 23 May 2022 22:39:37.973 * +slave slave 10.0.0.100:6379 10.0.0.100 6379 @ mymaster 10.0.0.102 6379
1466:X 23 May 2022 22:39:41.041 # +sdown slave 10.0.0.100:6379 10.0.0.100 6379 @ mymaster 10.0.0.102 6379

1.3.9、恢複故障的原master重新加入redis叢集 

#重新其他原master的redis服務，會發現會自動指向新的master
root@node1:~# systemctl restart redis
root@node1:~# grep ^replicaof /apps/redis/etc/redis.conf
replicaof 10.0.0.102 6379

#在原master上觀察狀态
root@node1:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:10.0.0.102
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:427125
slave_repl_offset:427125
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:9e7e93af69bee27494f2592f2f45ae5475583f8a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:427125
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:387282
repl_backlog_histlen:39844

root@node1:~# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.102:6379,slaves=2,sentinels=3

#在回到新master節點觀察狀态
root@node3:~# redis-cli -a wm521314
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:master
connected_slaves:2
slave0:ip=10.0.0.101,port=6379,state=online,offset=447650,lag=0
slave1:ip=10.0.0.100,port=6379,state=online,offset=447934,lag=0
master_failover_state:no-failover
master_replid:9e7e93af69bee27494f2592f2f45ae5475583f8a
master_replid2:9935b9865c7268c1aac772d2ed2a0deea625cdb5
master_repl_offset:447934
second_repl_offset:182386
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:447934
#在新主檢視一下日志
root@node3:~# tail -f /apps/redis/log/sentinel.log
........
1466:X 23 May 2022 22:39:41.041 # +sdown slave 10.0.0.100:6379 10.0.0.100 6379 @ mymaster 10.0.0.102 6379
1466:X 23 May 2022 22:56:29.257 # -sdown slave 10.0.0.100:6379 10.0.0.100 6379 @ mymaster 10.0.0.102 6379

一鍵安裝redis的腳本 

root@node1:~# vim install_redis.sh
#!/bin/bash

SRC_DIR=/usr/local/src
COLOR="echo -e \\033[01;31m"
END='\033[0m'
CPUS=`lscpu | awk '/^CPU\(s\)/{print $2}'`

URL='https://download.redis.io/releases/'
VERSION=redis-6.2.7
PASSWORD=wm521314
INSTALL_DIR=/apps/redis

os() {
    if grep -Eqi "CentOS" /etc/issue || grep -Eq "CentOS" /etc/*-release;then
        rpm -q redhat-lsb-core &> /dev/null || { ${COLOR}"安裝lsb_release工具"${END};yum -y install redhat-lsb-core &> /dev/null; }
    fi
    OS_ID=`lsb_release -is`
}

install() {
    if [ ${OS_ID} == "CentOS" ] &> /dev/null;then
        yum -y install gcc jemalloc-devel || { ${COLOR}"安裝軟體包失敗，請檢查網絡配置"${END}; exit; }
        rpm -q wget &> /dev/null || yum -y install wget &> /dev/null
    else
        apt -y install make gcc libjemalloc-dev || { ${COLOR}"安裝軟體包失敗，請檢查網絡配置"${END}; exit; }
    fi

    cd ${SRC_DIR}
    wget ${URL}${VERSION}.tar.gz || { ${COLOR}"Redis源碼下載下傳失敗"${END}; exit; }
    tar xf ${VERSION}.tar.gz
    cd ${VERSION}
    make -j ${CPUS} PREFIX=${INSTALL_DIR} install && ${COLOR}"Redis編譯安裝完成"${END} || { ${COLOR}"Redis編譯安裝失敗"${END}; exit; }
    ln -s ${INSTALL_DIR}/bin/redis-* /usr/bin/
    mkdir -p ${INSTALL_DIR}/{etc,log,data,run}
    cp redis.conf ${INSTALL_DIR}/etc/
    sed -i -e 's/bind 127.0.0.1.*/bind 0.0.0.0/' -e "/# requirepass/a requirepass ${PASSWORD}" -e "/^dir .*/c dir ${INSTALL_DIR}/data/" -e "/logfile .*/c logfile ${INSTALL_DIR}/log/redis-6379.log" -e "/^pidfile .*/c pidfile ${INSTALL_DIR}/run/redis_6379.pid" ${INSTALL_DIR}/etc/redis.conf

    if id redis &> /dev/null ;then
        ${COLOR}"Redis使用者已經存在"${END}
    else
        useradd -r -s /sbin/nologin redis
        ${COLOR}"Redis使用者建立成功"${END}
    fi

    chown -R redis.redis ${INSTALL_DIR}

    if [ ${OS_ID} == "CentOS" ] &> /dev/null;then
        echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.d/rc.local
        chmod +x /etc/rc.d/rc.local
    else
        cat >> /lib/systemd/system/rc-local.service <<-EOF
[Install]
WantedBy=multi-user.target
EOF
        echo '#!/bin/bash' > /etc/rc.local
        echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
        chmod +x /etc/rc.local
    fi
    
    cat > /lib/systemd/system/redis.service <<-EOF
[Unit]
Description=Redis persistent key-value database
After=network.target

[Service]
ExecStart=${INSTALL_DIR}/bin/redis-server ${INSTALL_DIR}/etc/redis.conf --supervised systemd
ExecStop=/bin/kill -s QUIT \$MAINPID
#Type=notify                                                                                                                                                                                                    
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
EOF
    systemctl daemon-reload
    systemctl enable --now redis &> /dev/null && ${COLOR}"Redis服務啟動成功,Redis資訊如下:"${END} || { ${COLOR}"Redis服務啟動失敗"${END};exit; }
    sleep 1
    redis-cli -a ${PASSWORD} INFO Server 2> /dev/null
}

main() {
    os
    install
}

main
Redis 伺服器 redis哨兵的實作 redis主從複制
上一篇: linux 擷取伺服器清單,如何在Linux中擷取CPU資訊
下一篇: tensorflow源碼分析（八）- 列印log

繼續閱讀