參考文檔：https://support.huaweicloud.com/dpmg-kunpengsdss/kunpengcephansible_04_0010.html
1、基礎環境
| ip | 主機名 | 系統版本 | 元件 | 資料盤 |
| 172.31.40.204 | ceph01 | centos7.9 | mons,osds,mgrs,mdss,clients,rgws | /dev/sdb |
| 172.31.40.205 | ceph02 | centos7.9 | mons,osds,mgrs,mdss,clients,rgws,grafana-server | /dev/sdb |
| 172.31.40.206 | ceph03 | centos7.9 | mons,osds,mgrs,mdss,clients,rgws | /dev/sdb |
需要執行的操作有做ceph01到ceph01、ceph02、ceph03的免密操作，同時三個節點需要做時間同步、關閉防火牆、selinux
三台機器安裝yum源
CentOS-Base.repo
[[email protected] yum.repos.d]# cat CentOS-Base.repo
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client. You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#
[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#released updates
[updates]
name=CentOS-$releasever - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
epel.repo
[[email protected] yum.repos.d]# cat epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
docker-ce.repo
[[email protected] yum.repos.d]# cat docker-ce.repo
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-test]
name=Docker CE Test - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-test-debuginfo]
name=Docker CE Test - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-test-source]
name=Docker CE Test - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-nightly]
name=Docker CE Nightly - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-nightly-debuginfo]
name=Docker CE Nightly - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-nightly-source]
name=Docker CE Nightly - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
三台機器清理緩存，并生成新的緩存
yum clean all && yum makecache
2、需要的軟體包
在github中下載下傳4版本的最新tar包，我下載下傳的是ceph-ansible-4.0.62.tar.gz
特别注意：centos7最高也隻能使用4版本，因為5和6版本需要centOS8
可以通過一下指令檢視系統版本是否滿足：
grep "This version is supported on RHEL" all.yml.sample
下載下傳tar包後，解壓到對應目錄
mkdir /root/ceph-ansible
tar -zxvf ceph-ansible-4.0.62.tar.gz -C /root/ceph-ansible
3、部署ceph叢集（隻需在ceph01節點上執行）
3.1、修改group_vars檔案夾中的内容
cd /root/ceph-ansible/ceph-ansible-4.0.62/group_vars
for file in *;do cp $file ${file%.*};done
主要修改兩個檔案all.yml 與osds.yml
all.yml全部内容為：
---
dummy:
configure_firewall: False
ceph_origin: repository
ceph_origin: repository
ceph_repository: community
ceph_mirror: http://mirrors.aliyun.com/ceph
ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc
ceph_stable_release: nautilus
ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}"
public_network: "172.31.0.0/16"
cluster_network: "172.31.0.0/16"
monitor_interface: eth0
osd_auto_discovery: true
dashboard_admin_password: [email protected]
grafana_admin_password: [email protected]
osd_objectstore: bluestore
radosgw_interface: eth0
pg_autoscale_mode: True
ceph_conf_overrides:
mon:
mon_allow_pool_delete: true
osds.yml修改内容為：
3.2、修改docker的配置
備注：預設為會安裝docker1.13版本，可能會造成docker啟動不成功，現修改docker為社群版本
cd /root/ceph-ansible/ceph-ansible-4.0.62/roles/ceph-container-engine/vars
修改RedHat.yml檔案為：
3.3、修改site.yml配置
cd /root/ceph-ansible/ceph-ansible-4.0.62
cp site.yml.sample site.yml
修改site.yml内容（注釋一些元件）
3.4、建立hosts檔案，并指定伺服器負載元件情況
cd /root/ceph-ansible/ceph-ansible-4.0.62
touch hosts
cat > /root/ceph-ansible/ceph-ansible-4.0.62/hosts << EOF
[mons]
172.31.40.204
172.31.40.205
172.31.40.206
[osds]
172.31.40.204
172.31.40.205
172.31.40.206
[mgrs]
172.31.40.204
172.31.40.205
172.31.40.206
[mdss]
172.31.40.204
172.31.40.205
172.31.40.206
[clients]
172.31.40.204
172.31.40.205
172.31.40.206
[rgws]
172.31.40.204
172.31.40.205
172.31.40.206
[grafana-server]
172.31.40.205
EOF
4、部署ceph
執行指令(在ceph01節點上)
pip install -r requirements.txt
#如果以上執行報錯，就手動安裝ansible
yum -y install ansible
#注意ansible版本情況
yum install -y python-netaddr
#安裝ceph
ansible-playbook -i hosts site.yml
#如果報錯，就是缺少元件， 對應安裝即可
安裝完成後，執行ceph -s可能會有一些警告，可以通過一下指令清除
ceph osd pool set <pool-name> pg_autoscale_mode on #<pool-name> 根據實際的名字确定
ceph config set mon auth_allow_insecure_global_id_reclaim false
ceph crash archive-all
如果要新增mon節點：
ansible-playbook -i hosts infrastructure-playbooks/add-mon.yml
如果删除叢集可用一下指令
ansible-playbook -i hosts infrastructure-playbooks/purge-cluster.yml
4、監控與dashboard
4.1、dashboard:
https://172.31.40.204:8443/#/dashboard
4.2監控：
4.2.1grafana
https://172.31.40.204:3000/?orgId=1
4.2.2 prometheus
http://172.31.40.204:9092/graph
4.2.3 alertmanager
http://172.31.40.204:9093/#/alerts