天天看點
傳回

Etcd基本部署與使用

etcd是什麼

根據etcd官網的介紹：

A distributed, reliable key-value store for the most critical data of a distributed system。分布式，可靠的鍵值存儲，用于分布式系統中最關鍵的資料。

在分布式系統中，各種服務配置資訊的管理共享和服務發現是一個很基本也是很重要的問題，無論你調用服務還是排程容器，都需要知道對應的服務執行個體和容器節點位址資訊。etcd 就是這樣一款實作了中繼資料資訊可靠存儲的元件。

etcd 可集中管理配置資訊。服務端将配置資訊存儲于 etcd，用戶端通過 etcd 得到服務配置資訊，etcd 監聽配置資訊的改變，發現改變通知用戶端。

而 etcd 滿足 CAP 理論中的 CP（一緻性和分區容錯性） 名額，由此我們知道，etcd 解決了分布式系統中一緻性存儲的問題。

二進制安裝 

ETCD_VER=v3.4.4
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GITHUB_URL}
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /usr/local/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /usr/local/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local
rm -f /usr/local/etcd-${ETCD_VER}-linux-amd64.tar.gz
ln /usr/local/etcd-${ETCD_VER}-linux-amd64/etcd  /usr/local/bin/etcd
ln /usr/local/etcd-${ETCD_VER}-linux-amd64/etcdctl  /usr/local/bin/etcdctl
etcd --version
etcdctl version

初始化etcd叢集

在每個台機器上設定etcd叢集成員參數：

TOKEN=cr7-etcd
CLUSTER_STATE=new
NAME_1=etcd1
NAME_2=etcd2
NAME_3=etcd3
HOST_1=192.168.1.91
HOST_2=192.168.1.92
HOST_3=192.168.1.93
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

分别在每台機器上運作初始化ectd叢集，初始化成功後會在運作該指令的目錄下生成data.etcd目錄，用于存放etcd節點相關資訊，隻要該目錄存在，停止後可以重新用該指令啟動：

# etcd1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
# etcd2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
# etcd3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

背景啟動可以加上：

nohup .... 2>&1 &

參數說明：

  • 預設2379端口是給client連接配接用的，而2380則在etcd叢集各個節點之間互動用的。
  • --name：etcd 叢集中的節點名，這裡可以随意，友善區分且不重複即可。
  • --listen-peer-urls：監聽用于節點之間通信的url，可監聽多個，叢集内部将通過這些url進行資料互動(如選舉、資料同步等)。
  • --initial-advertise-peer-urls：建議用于節點之間通信的 url，節點間将以該值進行通信。
  • --listen-client-urls：監聽用于用戶端通信的url，同樣可以監聽多個。
  • --advertise-client-urls：建議使用的用戶端通信url，該值用于etcd代理或etcd成員與etcd節點通信。
  • --initial-cluster-token：節點的token值，設定該值後叢集将生成唯一ID，并為每個節點也生成唯一ID。當使用相同配置檔案再啟動一個叢集時，隻要該token值不一樣，etcd叢集就不會互相影響。
  • --initial-cluster：叢集中所有的 initial-advertise-peer-urls的合集。
  • --initial-cluster-state：new，建立叢集的标志。existing辨別新加點加入叢集。

環境變量設定

編輯/etc/profile，将etcd叢集成員參數寫入環境變量：

export ETCDCTL_API=3
HOST_1=192.168.1.91
HOST_2=192.168.1.92
HOST_3=192.168.1.93
export ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
#編輯完成後
source /etc/profile

檢查etcd叢集狀态 

#檢視etcd叢集成員
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS member list  --write-out=table
+------------------+---------+-------+--------------------------+--------------------------+------------+
|        ID        | STATUS  | NAME  |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+-------+--------------------------+--------------------------+------------+
| 597745dd5a1f190a | started | etcd3 | http://192.168.1.93:2380 | http://192.168.1.93:2379 |      false |
| 6a3e67577dbd3de0 | started | etcd2 | http://192.168.1.92:2380 | http://192.168.1.92:2379 |      false |
| d2b20b971efea8ef | started | etcd1 | http://192.168.1.91:2380 | http://192.168.1.91:2379 |      false |
+------------------+---------+-------+--------------------------+--------------------------+------------+
#檢查叢集健康狀态
[root@etcd1 ~]# etcdctl endpoint health --cluster --endpoints=$ENDPOINTS
http://192.168.1.91:2379 is healthy: successfully committed proposal: took = 7.289365ms
http://192.168.1.92:2379 is healthy: successfully committed proposal: took = 13.173445ms
http://192.168.1.93:2379 is healthy: successfully committed proposal: took = 14.812638ms
#檢視叢集詳細狀态
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.91:2379 | d2b20b971efea8ef |   3.4.4 |   25 kB |     false |      false |         8 |         64 |                 64 |        |
| 192.168.1.92:2379 | 6a3e67577dbd3de0 |   3.4.4 |   25 kB |      true |      false |         8 |         64 |                 64 |        |
| 192.168.1.93:2379 | 597745dd5a1f190a |   3.4.4 |   25 kB |     false |      false |         8 |         64 |                 64 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

寫入資料 

etcdctl --endpoints=$ENDPOINTS put foo "Hello World"

讀取資料 

[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS get foo
foo
Hello World

通過字首擷取資料：

#分别插入web1，web2，web3 三條資料
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS put web1 value1
OK
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS put web2 value2
OK
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS put web3 value3
OK
#通過web字首擷取這三條資料
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS get web --prefix
web1
value1
web2
value2
web3
value3

删除資料 

[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS  del foo
1 #删除了1條資料
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS  del web --prefix
3 #删除了3條資料

事務寫入 

#先插入一條資料
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS put user1 bad
OK
#開啟事務
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS txn --interactive
#輸入判斷條件，兩次回車
compares:
value("user1") = "good"
#如果user1 = good，則執行del user1
success requests (get, put, del):
del user1
#如果user1 != good，則執行put user1 verygood
failure requests (get, put, del):
put user1 verygood
FAILURE
OK
#由于user1原先不等于good，是以執行put user1 verygood
[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS get user1
user1
verygood  #現在user1的值已經被改為verygood

監聽

watch用于擷取監聽資訊的更改，并且支援持續地監聽。在視窗1開啟監聽：

[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS watch

另外開啟一個視窗2寫入資料：

etcdctl --endpoints=$ENDPOINTS put stock1 1000

此時視窗1會收到更新資訊：

stock1
PUT
stock1
1000

也支援字首監聽：

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

租約

lease用于設定key的TTL時間。

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)
#建立資料，并指定lease
etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
#此時還可以擷取到資料
etcdctl --endpoints=$ENDPOINTS get sample
#重置租約時間到原先指定的300s，會重複重新整理
etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
#立即釋放
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
#租約到期或者直接revoke，就擷取不到這個key了
etcdctl --endpoints=$ENDPOINTS get sample

分布式鎖

視窗1給key1加鎖：

[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS lock key1
key1/28ef77fb1b464b49

視窗2也想給key1加鎖，此時會卡住，直到視窗1釋放鎖以後，視窗2才能給key1加鎖：

[root@etcd1 ~]# etcdctl --endpoints=$ENDPOINTS lock key1

快照

snapshot隻能指定其中一個etcd節點：

[root@etcd1 snap]# ENDPOINTS=$HOST_1:2379
[root@etcd1 snap]# etcdctl --endpoints=$ENDPOINTS snapshot save my.db
{"level":"info","ts":1614828932.1646311,"caller":"snapshot/v3_snapshot.go:110","msg":"created temporary db file","path":"my.db.part"}
{"level":"info","ts":1614828932.166052,"caller":"snapshot/v3_snapshot.go:121","msg":"fetching snapshot","endpoint":"192.168.1.91:2379"}
{"level":"info","ts":1614828932.1835077,"caller":"snapshot/v3_snapshot.go:134","msg":"fetched snapshot","endpoint":"192.168.1.91:2379","took":0.018725853}
{"level":"info","ts":1614828932.1836193,"caller":"snapshot/v3_snapshot.go:143","msg":"saved","path":"my.db"}
Snapshot saved at my.db
#快照存放在data.etcd/member/snap
[root@etcd1 snap]# ls
db  my.db
[root@etcd1 snap]# etcdctl --write-out=table --endpoints=$ENDPOINTS snapshot status my.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 4c02b18b |       36 |         44 |      25 kB |
+----------+----------+------------+------------+

添加&删除成員

删除老成員，添加新成員（在原先etcd節點上操作）：

# 擷取成員ID
export ETCDCTL_API=3
HOST_1=192.168.1.91
HOST_2=192.168.1.92
HOST_3=192.168.1.93
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 member list
# 移除成員
MEMBER_ID=278c654c9a6dfd3b #移除node3
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 \
 member remove ${MEMBER_ID}
# 添加新成員 (etcd4)
export ETCDCTL_API=3
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_4=etcd-node-4
HOST_1=192.168.1.91
HOST_2=192.168.1.92
HOST_4=192.168.1.94 # new member
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379 \
 member add ${NAME_4} \
 --peer-urls=http://${HOST_4}:2380

在新etcd節點操作：

#如果新成員在相同的磁盤上啟動，確定移除data目錄
TOKEN=cr7-etcd
CLUSTER_STATE=existing  #加入新叢集
NAME_1=etcd1
NAME_2=etcd2
NAME_4=etcd4
HOST_1=192.168.1.91
HOST_2=192.168.1.92
HOST_4=192.168.1.94 # new member
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_4}=http://${HOST_4}:2380
THIS_NAME=${NAME_4}
THIS_IP=${HOST_4}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

認證

root 是 etcd 的超級管理者，擁有 etcd 的所有權限，在開啟角色認證之前為們必須要先建立好 root 使用者。還需要注意的是 root 使用者必須擁有 root 的角色,允許在 etcd 的所有操作。（有一個特殊使用者root，一個特殊角色root。）

#建立root使用者，root使用者自動有最高root權限
[root@etcd1 ~]# etcdctl  --endpoints=$ENDPOINTS user add root
#設定密碼123456
Password of root: 
Type password of root again for confirmation: 
User root created
#建立一個普通權限，可以對foo這key進行讀寫操作
#建立一個role
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} role add user1-role
Role user1-role created
#給role賦予權限
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} role grant-permission user1-role  readwrite foo
Role user1-role updated
#檢視建立的role
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} role get user1-role
Role user1-role
KV Read:
        foo
KV Write:
        foo
        
#建立使用者并關聯權限
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} user add user1
#設定密碼123
Password of user1: 
Type password of user1 again for confirmation: 
User user1 created
#将user1-role的權限關聯到user1
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} user grant-role user1 user1-role
Role user1-role is granted to user user1
#啟用認證
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} auth enable
Authentication Enabled
#此時不指定使用者将無法進行任何操作
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} put foo bar
{"level":"warn","ts":"2021-03-04T22:53:30.740+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-b379b941-6eec-46a3-87ac-76417ae9ea5f/192.168.1.91:2379","attempt":0,"error":"rpc error: code = InvalidArgument desc = etcdserver: user name is empty"}
Error: etcdserver: user name is empty
#使用使用者user1可以對foo進行讀寫操作
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} put foo bar --user=user1:123
OK
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} get foo --user=user1:123
foo
bar
#但是user1無法對其他key進行操作
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} put foo2 bar2 --user=user1:123
{"level":"warn","ts":"2021-03-04T22:53:57.474+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-f67861bf-ce78-48b2-90a6-1f91ce508936/192.168.1.91:2379","attempt":0,"error":"rpc error: code = PermissionDenied desc = etcdserver: permission denied"}
Error: etcdserver: permission denied
#使用root使用者關閉認證
[root@etcd1 ~]# etcdctl --endpoints=${ENDPOINTS} auth disable --user=root:123456
Authentication Disabled
存儲 NoSQL 排程 容器 存儲容器 nosql存儲 nosql資料存儲 nosql列存儲 nosql存儲列存儲
上一篇: es實戰-使用IK分詞器進行詞頻統計1. IK 分詞統計代碼2. 擷取詞雲 TopN 個詞3. IK 代碼淺析4. 進行詞雲展示5. 總結
下一篇: 鄭葉來：為什麼華為雲的“三不”很重要

繼續閱讀