Codis集群搭建

文章目录

• 安装部署
• • 1. 下载程序
  • • 1)下载golang语言程序包,
    • 2)下载codis3.2.1（无需编译）
  • 2. 安装程序
  • • 1）安装java
    • 2）安装golang
    • 3)安装zookeeper
    • 4)安装codis
  • 3. 配置程序
  • • 1) 配置zookeeper
    • 2) 配置codis-server,213/214同样配置
    • 3) 配置redis-sentinel,213/214同样配置
    • 4) 配置codis-proxy,213/214同样配置
    • 5) 配置codis-dashboard,只需214配置
    • 5) 配置codis-fe,只需214配置
  • 4.web配置
  • • 1）添加codis-proxy地址和端口
    • 2）添加redis-server(也是codis-server)地址和端口
    • 3）配置sentinel的地址和端口（SYNC飘红的话点击同步下即可）
  • 5.启动环境
  • 6.故障解决
  • 【关闭密码校验】

安装部署

1. 下载程序

1)下载golang语言程序包,

codis3.2.0要求至少是1.7.5或1.8.1以上版本的,可下载最新版，当前使用1.8.1版本.

中文网地址：https://studygolang.com/dl/golang

2)下载codis3.2.1（无需编译）

GIT地址：https://github.com/CodisLabs/codis/releases

2. 安装程序

1）安装java

下载jdk1.8安装解压配置环境变量：
# vim ~/.bashrc  <增加如下配置>
#jdk#
export JAVA_HOME=/home/java
export JRE_HOME=$JAVA_HOME/jre
export CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
           

2）安装golang

# tar -zxvf go1.8.1.linux-amd64.tar.gz
# mv go /usr/local
# ln -sf /usr/local/go/bin/* /usr/bin/
# go version
           

3)安装zookeeper

下载zookeeper解压，配置zoo.cfg即可：
tickTime=2000
initLimit=10
syncLimit=5
maxSessionTimeout=180000
minSessionTimeout=180000
maxClientCnxns=1000
zookeeper.leaderServes=yes
zookeeper.service.jute.maxbuffer=1024KB
dataDir=/home/zookeeper/data
dataLogDir=/home/zookeeper/data/logs
clientPort=2181
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
isObserver=false
zookeeper.oom_heap_dump_enabled=true
zookeeper.oom_heap_dump_dir=/home/zookeeper/data
zookeeper.java_additional_options=-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
zookeeper.oom_sigkill_enabled=true
zookeeper.java_heapsize=2048MB
electionPort=2888
quorumPort=3888
zookeeper.sasl=false
observers=
server.1=testai-1:2888:3888
server.2=testai-2:2888:3888
           

4)安装codis

# tar -zxvf codis3.2.0-go1.8.1-linux.tar.gz
# mv codis3.2.0-go1.8.1-linux /usr/local
# ln -sf codis3.2.0-go1.8.1-linux codis
           

3. 配置程序

1) 配置zookeeper

2) 配置codis-server,213/214同样配置

注意:codis-server就是redis-server程序,属于codis优化版本,配合codis集群使用.
【创建redis数据目录,配置文件目录,日志目录】
# mkdir -p /home/jetflow/codis/data/config/
# mkdir -p /home/jetflow/codis/data/logs/
【创建主库的配置文件,暂时只配置这些,其他先默认】
# vim /home/codis/data/config/redis_6379.conf
>>>>>>
#允许后台运行
daemonize yes
#设置端口,最好是非默认端口
port 6379
#绑定登录IP,安全考虑,最好是内网
bind *
#命名并指定当前redis的PID路径,用以区分多个redis
pidfile "/home/codis/data/config/redis_6379.pid"
#命名并指定当前redis日志文件路径
logfile "/home/codis/data/logs/redis_6379.log"
#指定RDB文件名,用以备份数据到硬盘并区分不同redis,当使用内存超过可用内存的45%时触发快照功能
dbfilename "dump_6379.rdb"
#指定当前redis的根目录,用来存放RDB/AOF文件
dir "/home/codis/data"
#当前redis的认证密钥,redis运行速度非常快,这个密码要足够强大,
#所有codis-proxy集群相关的redis-server认证密码必须全部一致
requirepass "Qax1212!%&@"
#当前redis的最大容量限制,建议设置为可用内存的45%内,最高能设置为系统可用内存的95%,
#可用config set maxmemory 去在线修改,但重启失效,需要使用config rewrite命令去刷新配置文件
#注意,使用codis集群,必须配置容量大小限制,不然无法启动
maxmemory 100000kb
#LRU的策略,有四种,看情况选择
maxmemory-policy allkeys-lru
#如果做故障切换，不论主从节点都要填写密码且要保持一致
masterauth "Qax1212!%&@"
# Generated by CONFIG REWRITE
<<<<<<
【创建从库的配置文件,暂时只配置这些,其他先默认】
# vim /home/codis/data/config/redis_6380.conf
>>>>>>
#允许后台运行
daemonize yes
#设置端口,最好是非默认端口
port 6380
#绑定登录IP,安全考虑,最好是内网
bind *
#命名并指定当前redis的PID路径,用以区分多个redis
pidfile "/home/codis/data/config/redis_6380.pid"
#命名并指定当前redis日志文件路径
logfile "/home/codis/data/logs/redis_6380.log"
#指定RDB文件名,用以备份数据到硬盘并区分不同redis,当使用内存超过可用内存的45%时触发快照功能
dbfilename "dump_6380.rdb"
#指定当前redis的根目录,用来存放RDB/AOF文件
dir "/home/codis/data"
#当前redis的认证密钥,redis运行速度非常快,这个密码要足够强大
#所有codis-proxy集群相关的redis-server认证密码必须全部一致
requirepass "Qax1212!%&@"
#当前redis的最大容量限制,建议设置为可用内存的45%内,最高能设置为系统可用内存的95%,
#可用config set maxmemory 去在线修改,但重启失效,需要使用config rewrite命令去刷新配置文件
#注意,使用codis集群,必须配置容量大小限制,不然无法启动
maxmemory 100000kb
#LRU的策略,有四种,看情况选择
maxmemory-policy allkeys-lru
#如果做故障切换，不论主从节点都要填写密码且要保持一致
masterauth "Qax1212!%&@"
#配置主节点信息(213/214部署记得切换IP)
# Generated by CONFIG REWRITE
slaveof 10.47.85.214 6379
<<<<<<
注意：除了端口号不同带来的文件名不同.实际上从库配置只是多了最后一行,指定了主库地址

【启动codis集群：（其实codis-server就是redis-server）】
# /usr/local/codis/codis-server /home/codis/data/config/redis_6379.conf
# /usr/local/codis/codis-server /home/codis/data/config/redis_6380.conf
           

3) 配置redis-sentinel,213/214同样配置

正确来说,redis-sentinel是要配置主从架构才能生效,但是在codis集群中并不一样,因为他的配置由zookeeper来维护,所以,这里codis使用的redis-sentinel只需要配置一些基本配置就可以了.

#配置放到redis数据目录的配置文件目录

# vim /home/codis/data/config/sentinel.conf
>>>>>>
bind 0.0.0.0
protected-mode no
port 26379
dir "/home/codis/data"
pidfile "/home/codis/data/config/sentinel_26379.pid"
logfile "/home/codis/data/logs/sentinel_26379.log"
daemonize yes
<<<<<<

#修改somaxconn（如果报错需要操作）
echo 512 > /proc/sys/net/core/somaxconn

#然后启动（redis-sentinel需要从redis/src下拷贝到codis下）
/usr/local/codis/redis-sentinel /home/codis/data/config/sentinel.conf --sentinel

#验证
/usr/local/codis/redis-cli -p 26379 -c info Sentinel
↓↓↓↓↓↓↓↓↓打印信息↓↓↓↓↓↓↓↓↓
# Sentinel
sentinel_masters:3
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=codis-test1-3,status=ok,address=10.0.2.7:6380,slaves=1,sentinels=3
master1:name=codis-test1-1,status=ok,address=10.0.2.5:6379,slaves=1,sentinels=3
master2:name=codis-test1-2,status=ok,address=10.0.2.6:6379,slaves=1,sentinels=3
↑↑↑↑↑↑↑↑↑打印信息↑↑↑↑↑↑↑↑↑
注意:没有配置好codis-dashboard会没最后那几行,因为还不受zookeeper控制,所以是正常的,配置好之后才会自动加载进来.
           

4) 配置codis-proxy,213/214同样配置

这个是codis集群的核心,实际上他也没配主从架构,配置也是从zookeeper拿来用的。

#配置很多,先生成默认的配置文件
/usr/local/codis/codis-proxy --default-config | tee /usr/local/codis/proxy.conf
#然后把配置放到redis数据目录的配置文件目录,再更改关键位置,其他默认即可
vim /usr/local/codis/proxy.conf
【配置要点】
>>>>>>
#项目名称,会登记在zookeeper里,如果你想一套zookeeper管理多套codis,就必须区分好
product_name = "codis-test1"
# 设置登录dashboard的密码(与真实redis中requirepass一致)
product_auth = "Qax1212!%&@"
#客户端(redis-cli)的登录密码(与真实redis中requirepass不一致),是登录codis的密码
session_auth = "Qax1212!%&@"
#管理的端口,0.0.0.0即对所有ip开放,基于安全考虑,可以限制内网
admin_addr = "0.0.0.0:11080"
#用那种方式通信,假如你的网络支持tcp6的话就可以设别的
proto_type = "tcp4"
#客户端(redis-cli)访问代理的端口,0.0.0.0即对所有ip开放
proxy_addr = "0.0.0.0:19000"
#外部配置存储类型,我们用的就是zookeeper,当然也是还有其他可以支持,这里不展开说
jodis_name = "zookeeper"
#配置zookeeper的连接地址,这里是三台就填三台
jodis_addr = "10.47.85.214:2181,10.47.85.213:2181"
#zookeeper的密码,假如有的话
jodis_auth = ""
#codis代理的最大连接数,默认是1000,并发大要调大
proxy_max_clients = 100000
#假如并发太大,你可能需要调这个pipeline参数,大多数情况默认10000就够了
session_max_pipeline = 100000
#并发大也可以改以下参数
backend_max_pipeline = 204800
session_recv_bufsize = "256kb"
session_recv_timeout = "0s"
<<<<<<
【全量配置】
>>>>>>
##################################################
#                                                #
#                  Codis-Proxy                   #
#                                                #
##################################################

# Set Codis Product Name/Auth.
product_name = "codis-test1"
product_auth = "Qax1212!%&@"

# Set auth for client session
#   1. product_auth is used for auth validation among codis-dashboard,
#      codis-proxy and codis-server.
#   2. session_auth is different from product_auth, it requires clients
#      to issue AUTH <PASSWORD> before processing any other commands.
session_auth = "Qax1212!%&@"

# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:11080"

# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
proto_type = "tcp4"
proxy_addr = "0.0.0.0:19000"

# Set jodis address & session timeout
#   1. jodis_name is short for jodis_coordinator_name, only accept "zookeeper" & "etcd".
#   2. jodis_addr is short for jodis_coordinator_addr
#   3. proxy will be registered as node:
#        if jodis_compatible = true (not suggested):
#          /zk/codis/db_{PRODUCT_NAME}/proxy-{HASHID} (compatible with Codis2.0)
#        or else
#          /jodis/{PRODUCT_NAME}/proxy-{HASHID}
jodis_name = "zookeeper"
jodis_addr = "10.47.85.214:2181,10.47.85.213:2181"
jodis_timeout = "20s"
jodis_compatible = false

# Set datacenter of proxy.
proxy_datacenter = ""

# Set max number of alive sessions.
proxy_max_clients = 1000

# Set max offheap memory size. (0 to disable)
proxy_max_offheap_size = "1024mb"

# Set heap placeholder to reduce GC frequency.
proxy_heap_placeholder = "256mb"

# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
backend_ping_period = "5s"

# Set backend recv buffer size & timeout.
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"

# Set backend send buffer & timeout.
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"

# Set backend pipeline buffer size.
backend_max_pipeline = 20480

# Set backend never read replica groups, default is false
backend_primary_only = false

# Set backend parallel connections per server
backend_primary_parallel = 1
backend_replica_parallel = 1

# Set backend tcp keepalive period. (0 to disable)
backend_keepalive_period = "75s"

# Set number of databases of backend.
backend_number_databases = 16

# If there is no request from client for a long time, the connection will be closed. (0 to disable)
# Set session recv buffer size & timeout.
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"

# Set session send buffer size & timeout.
session_send_bufsize = "64kb"
session_send_timeout = "30s"

# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
# Set session pipeline buffer size.
session_max_pipeline = 10000

# Set session tcp keepalive period. (0 to disable)
session_keepalive_period = "75s"

# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
session_break_on_failure = false

# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
metrics_report_server = ""
metrics_report_period = "1s"

# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""

# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""
<<<<<<

#启动
/usr/local/codis/codis-proxy --ncpu=1 --config=/usr/local/codis/proxy.conf --log=/home/codis/data/logs/proxy.log &

#验证一下
ss -ntplu |grep codis-proxy
↓↓↓↓↓↓↓↓↓打印信息↓↓↓↓↓↓↓↓↓
tcp	LISTEN	0	512	*:19000	*:*	users:((""codis-proxy),pid=7488,fd=4))
tcp	LISTEN	0	512	:::11080	::::*	users:((""codis-proxy),pid=7488,fd=6))
↑↑↑↑↑↑↑↑↑打印信息↑↑↑↑↑↑↑↑↑
           

5) 配置codis-dashboard,只需214配置

dashboard属于管理配置codis集群信息的工具,配置完之后的配置信息会自动加载到zookeeper集群,即使这个服务挂了,配置都还在zookeeper上,所以不用考虑高可用,单点就足够了,大不了重新启动一下也不是特别麻烦,配置界面由codis-fe来实现.因为限制于zookeeper的命名空间,通常是和proxy一套配置.

#用程序来生成默认配置文件
/usr/local/codis/codis-dashboard --default-config | tee ./dashboard.conf
#然后我们把配置放到redis数据目录的配置文件目录,再更改关键位置,其他默认即可
vim /usr/local/codis/dashboard.conf
>>>>>>
【配置要点】
#外部配置存储类型,我们用的就是zookeeper,当然也是还有其他可以支持,这里不展开说
coordinator_name = "zookeeper"
#配置zookeeper的连接地址,这里是三台就填三台
coordinator_addr = "10.47.85.214:2181,10.47.85.213:2181"
#项目名称,会登记在zookeeper里,如果你想一套zookeeper管理多套codis,就必须区分好
product_name = "codis-test1"
#所有redis的登录密码(与真实redis中requirepass一致),因为要登录进去修改数据
product_auth = "Qax1212!%&@"
#codis-dashboard的通信端口,0.0.0.0表示对所有开放,最好使用内网地址
admin_addr = "0.0.0.0:18080"
#如果想要在codis集群在故障切换功能上执行一些脚本,可以配置以下两个配置
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""
<<<<<<
【全量配置】
>>>>>>
##################################################
#                                                #
#                  Codis-Dashboard               #
#                                                #
##################################################

# Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".
# Quick Start
coordinator_name = "zookeeper"
coordinator_addr = "10.47.85.214:2181,10.47.85.213:2181"
#coordinator_name = "zookeeper"
#coordinator_addr = "127.0.0.1:2181"

# Set Codis Product Name/Auth.
product_name = "codis-test1"
product_auth = "Qax1212!%&@"

# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:18080"

# Set arguments for data migration (only accept 'sync' & 'semi-async').
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"

# Set configs for redis sentinel.
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "30s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""
<<<<<<
#启动
/usr/local/codis/codis-dashboard --ncpu=1 --config=/usr/local/codis/dashboard.conf --log=/home/jetflow/codis/data/logs/codis_dashboard.log --log-level=WARN &
#验证
ss -ntplu |grep codis-dashboard
↓↓↓↓↓↓↓↓↓打印信息↓↓↓↓↓↓↓↓↓
tcp	LISTEN	0	512	:::18080	::::*	users:((""codis-dashboard),pid=27488,fd=5))
↑↑↑↑↑↑↑↑↑打印信息↑↑↑↑↑↑↑↑↑

注意：
异常退出会lock dashboard：解决办法
# /usr/local/codis/codis-admin --remove-lock --product=codis-test1 --zookeeper=127.0.0.1:2181
异常无法恢复则删除zookeeper数据：操作后需要管理页面重新配置集群
# /home/zookeeper/bin/zkCli.sh;rmr Jodis codis3
           

5) 配置codis-fe,只需214配置

这个是属于web界面操作codis-dashboard配置的工具,web代码文件在codis安装文件夹的目录下,具体是:/usr/local/codis/assets/这个目录.可多台dashboard共用.

这个工具本身不需要配置文件就能启动,只需要指定codis-dashboard的ip和端口就可以了,但是我为了方便管理,还是生成一个配置文件的好.

#生成配置文件,其实也就是codis-dashboard的ip和端口()
/usr/local/codis/codis-admin --dashboard-list --zookeeper=10.47.85.214:2181 > /usr/local/codis/codis.json
#把配置放到redis数据目录的配置文件目录
cat /usr/local/codis/codis.json 
[
    {
        "name": "codis-test1",
        "dashboard": "10.47.85.214:18080"
    }
]
#启动
/usr/local/codis/codis-fe --ncpu=1 --log=/home/codis/data/logs/fe.log --log-level=WARN --dashboard-list=/usr/local/codis/codis.json --listen=0.0.0.0:8090 &
#web地址
http://10.47.85.214:8090/
           

4.web配置

1）添加codis-proxy地址和端口

方法：输入IP：Port后点击“New Proxy”

2）添加redis-server(也是codis-server)地址和端口

3）配置sentinel的地址和端口（SYNC飘红的话点击同步下即可）

### 4）测试集群

5.启动环境

$ cat startcodis.sh
echo "rmr /jodis"|/home/jetnk/zookeeper/bin/zkCli.sh -server 10.194.22.202:3181
echo "rmr /codis3"|/home/jetnk/zookeeper/bin/zkCli.sh -server 10.194.22.202:3181
echo "rmr /codis-test1"|/home/jetnk/zookeeper/bin/zkCli.sh -server 10.194.22.202:3181
sleep 1
ps -ef|grep -w 'codis'|grep -v startcodis|awk '{print $2}'|xargs kill -9
sleep 2
/home/jetnk/codis/codis-server /home/jetnk/codis/data/config/redis_6379.conf
/home/jetnk/codis/codis-server /home/jetnk/codis/data/config/redis_6380.conf
sleep 2
/home/jetnk/codis/redis-sentinel /home/jetnk/codis/data/config/sentinel.conf --sentinel
sleep 2
/home/jetnk/codis/redis-cli -p 26379 -c info Sentinel
sleep 2
/home/jetnk/codis/codis-proxy --ncpu=1 --config=/home/jetnk/codis/proxy.conf --log=/home/jetnk/codis/data/logs/proxy.log &
sleep 2
ss -ntplu |grep codis-proxy
sleep 2
/home/jetnk/codis/codis-dashboard --ncpu=1 --config=/home/jetnk/codis/dashboard.conf --log=/home/jetnk/codis/data/logs/codis_dashboard.log --log-level=WARN &
sleep 2
ss -ntplu |grep codis-dashboard
sleep 2
/home/jetnk/codis/codis-fe --ncpu=1 --log=/home/jetnk/codis/data/logs/fe.log --log-level=WARN --dashboard-list=/home/jetnk/codis/codis.json --listen=0.0.0.0:8090 &


$ cat delcodispass.sh
echo "config set requirepass \"\""|/home/jetnk/codis/redis-cli -c -h 10.194.22.202 -p 6379 -a 'Qax1212!%&@'
echo "config set requirepass \"\""|/home/jetnk/codis/redis-cli -c -h 10.194.22.202 -p 6380 -a 'Qax1212!%&@'


$ cat recoverypass.sh
echo "config set requirepass \"Qax1212!%&@\""|/home/jetnk/codis/redis-cli -c -h 10.194.22.202 -p 6379 -a 'Qax1212!%&@'
echo "config set requirepass \"Qax1212!%&@\""|/home/jetnk/codis/redis-cli -c -h 10.194.22.202 -p 6380 -a 'Qax1212!%&@'
           

6.故障解决

新建proxy：
codis-admin --dashboard=10.10.11.214:18080 --create-proxy -x 10.10.11.213:11080
查询proxy状态：
codis-admin --dashboard=10.10.11.214:18080 --proxy-status
	[E] proxy-6 [T] 879cf157e6927be303a72cd07b4a04ef [A] testai-1:11080 [P] testai-1:19000
强制删除proxy：
codis-admin --dashboard=10.10.11.214:18080 --remove-proxy --token=9981febeefacebb218f09194f73436f8 --force
           

【关闭密码校验】

@ 修改配置
# vim config/redis_6379.conf
# requirepass "Qax1212!%&@"  # 注释此行
@ 临时关闭密码
# ./redis-cli -c  -h 10.10.11.214 -p 6379 -a ''
10.10.11.214:6379> config set requirepass ""
OK

如关闭无效，则增加下方操作：
修改以下文件(auth部分)，注释或者删除密码；
config/sentinel.conf
dashboard.conf
proxy.conf
t-redis.sh  # 密码改为 ''
替换命令：【:%s#Qax1212!%&@##g】