作者:吳業亮
部落格:wuyeliang.blog.csdn.net
一.架構及方案
- Ceph Block架構
2. Ceph iSCSI基本架構
iSCSI gateway的實作主要有TGT && LIO兩種方式。
-
TGT
TGT:Linux target framework,為建立、維護SCSI target 驅動(包括iSCSI、FC、SRP等)提供支援。
- 工作在使用者空間;
- 在Linux 2.6.38 版本後(含),核心內建了 Linux-IO Target。
在ceph內建原生iscsi之前,通常使用基于使用者空間的"scsi-target-utils"套件實作tgt。
基于centos7.x系列,安裝"scsi-target-utils"套件後,iscsi并不支援ceph rbd後端存儲(通過"tgtadm --lld iscsi --mode system --op show"檢視),主要原因是redhat針對套件屏蔽了支援ceph rbd後端存儲的代碼。
解決方案:
通過"rbd map xxx"将ceph rbd挂載到本地後,再通過iscsi tgt的"direct-store"模式釋出ceph塊存儲。
方案缺點:
"rbd map xxx"挂載ceph rbd是通過"ceph rbd kernel module"的形式,tgt在使用者空間實作,導緻釋出的ceph rbd在核心态與使用者态之間頻繁切換,影響性能。
-
LIO
LIO:Linux-IO Target,用軟體實作各種SCIS Target。
- 工作在核心空間;
- 支援較多傳輸協定,如Fibre Channel(Qlogic,linux3.5)、FCoE(linux3.0)、iSCSI(linux 3.1)、iSER (Mellanox InfiniBand,linux3.10), SRP (Mellanox InfiniBand,linux3.3), USB等;
- 對 iSCSI RFC 規範的支援非常好,包括完整的錯誤恢複都有支援;
- 從核心 3.17 開始引入使用者态後端支援,即 TCMU(Target Core Module in Userspace)
- 在Linux 2.6.38 版本後(含),核心內建了 Linux-IO Target。
本文主要介紹基于LIO的ceph原生iscsi 實作方式,LIO利用使用者空間直通(即TCMU)與ceph的librbd庫進行互動(tcmu-runner處理LIO TCM後端存儲的使用者空間端的守護程序,在核心之上多了一個使用者态的驅動層,這樣隻需要根據tcmu的标準來對接接口即可,而不用去直接與核心進行互動),并将rbd image暴露給iSCSI用戶端。
二、前提條件
1、版本
- Ceph Luminous 版本的叢集或者更新的版本
- RHEL/CentOS 7.5或者Linux kernel v4.16或者更新版本的核心
-
其他控制軟體
targetcli-2.1.fb47 or newer package
python-rtslib-2.1.fb68 or newer package
tcmu-runner-1.4.0 or newer package
ceph-iscsi-3.2 or newer package
2、針對ceph-mon或osd節點,并沒有特殊的iscsi-gateway參數選項,但降低若幹預設的檢測osd當機時間,可以有效降低initiator的連接配接逾時。可在ceph-mon節點修改ceph.conf檔案後分發到所有節點,如:
方法一:
# 新增參數
[osd]
osd client watch timeout = 15
osd heartbeat grace = 20
osd heartbeat interval = 5
重新開機服務
systemctl restart ceph.target
方法二:
指令修改方法
# ceph tell osd.* config set osd_client_watch_timeout 15
# ceph tell osd.* config set osd_heartbeat_grace 20
# ceph tell osd.* config set osd_heartbeat_interval 5
驗證方法:
[[email protected] ~]# ceph daemon osd.0 config get osd_client_watch_timeout
{
"osd_client_watch_timeout": "15"
}
[[email protected] ~]# ceph daemon osd.0 config get osd_heartbeat_grace
{
"osd_heartbeat_grace": "20"
}
[[email protected] ~]# ceph daemon osd.0 config get osd_heartbeat_interval
{
"osd_heartbeat_interval": "5"
}
三、自動化部署
環境
1、通過ceph-ansible安裝
# yum install ceph-ansible
其他ceph-ansible具體安裝參見
https://wuyeliang.blog.csdn.net/article/details/105344968
2、修改/etc/ansible/hosts,新增需要安裝iscsi-gateway的節點主機名
[iscsigws]
node1
node2
node3
3、安裝
# cd /usr/share/ceph-ansible
# ansible-playbook site.yml --limit iscsigws
4、檢視狀态
# ceph -s
cluster:
id: 8789218f-5fca-4727-9c74-aff3c2cebeab
health: HEALTH_OK
services:
mon: 3 daemons, quorum node1,node2,node3 (age 14m)
mgr: node2(active, since 14m), standbys: node1, node3
mds: cephfs:1 {0=node2=up:active} 2 up:standby
osd: 48 osds: 48 up (since 14m), 48 in (since 65m)
rgw: 3 daemons active (node1.rgw0, node2.rgw0, node3.rgw0)
tcmu-runner: 2 daemons active (node1:wyl/disk, node2:wyl/disk)
data:
pools: 8 pools, 2200 pgs
objects: 43.01k objects, 107 GiB
usage: 65 GiB used, 9.3 TiB / 9.4 TiB avail
pgs: 2200 active+clean
io:
client: 3.0 KiB/s rd, 2 op/s rd, 0 op/s wr
# gwcli ls
Warning: Could not load preferences file /root/.gwcli/prefs.bin.
o- / .................................................................................................... [...]
o- cluster .................................................................................... [Clusters: 1]
| o- ceph ....................................................................................... [HEALTH_OK]
| o- pools ..................................................................................... [Pools: 8]
| | o- .rgw.root ......................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 1536K]
| | o- cephfs_data ....................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 0.00Y]
| | o- cephfs_metadata ................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 1536K]
| | o- default.rgw.control ............................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 0.00Y]
| | o- default.rgw.log ................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 0.00Y]
| | o- default.rgw.meta ................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 384K]
| | o- rbd ................................................ [(x3), Commit: 0.00Y/3089638M (0%), Used: 192K]
| | o- wyl ........................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 17388864K]
| o- topology .......................................................................... [OSDs: 48,MONs: 3]
o- disks .................................................................................. [0.00Y, Disks: 0]
o- iscsi-targets .......................................................... [DiscoveryAuth: None, Targets: 0
5、後面具體的服務為rbd-target-api,可以通過systemctl來管理
# systemctl <start|stop|restart|reload> rbd-target-api
6、登入web建立iscsi裝置
7、服務解除安裝
cd /usr/share/ceph-ansible/
ansible-playbook purge_gateways.yml
四、手動部署環境
1、安裝軟體
配置源
[[email protected] yum.repos.d]# cat ceph-iscsi.repo
[ceph-iscsi]
name=ceph-iscsi noarch packages
baseurl=http://download.ceph.com/ceph-iscsi/3/rpm/el7/noarch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
type=rpm-md
[ceph-iscsi-source]
name=ceph-iscsi source packages
baseurl=http://download.ceph.com/ceph-iscsi/3/rpm/el7/SRPMS
enabled=0
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
type=rpm-md
[[email protected] yum.repos.d]# cat python-rtslib-dev.repo
[python-rtslib]
name=python-rtslib packages for $basearch
baseurl=https://2.chacra.ceph.com/r/python-rtslib/master/67eb1605c697b6307d8083b2962f5170db13d306/centos/7/flavors/default/$basearch
enabled=1
gpgcheck=0
type=rpm-md
[python-rtslib-noarch]
name=python-rtslib noarch packages
baseurl=https://2.chacra.ceph.com/r/python-rtslib/master/67eb1605c697b6307d8083b2962f5170db13d306/centos/7/flavors/default/noarch
enabled=1
gpgcheck=0
type=rpm-md
[python-rtslib-source]
name=python-rtslib source packages
baseurl=https://2.chacra.ceph.com/r/python-rtslib/master/67eb1605c697b6307d8083b2962f5170db13d306/centos/7/flavors/default/SRPMS
enabled=1
gpgcheck=0
type=rpm-md
[[email protected] yum.repos.d]# cat tcmu-runner-dev.repo
[tcmu-runner]
name=tcmu-runner packages for $basearch
baseurl=https://1.chacra.ceph.com/r/tcmu-runner/master/9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/$basearch
enabled=1
gpgcheck=0
type=rpm-md
[tcmu-runner-noarch]
name=tcmu-runner noarch packages
baseurl=https://1.chacra.ceph.com/r/tcmu-runner/master/9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/noarch
enabled=1
gpgcheck=0
type=rpm-md
[tcmu-runner-source]
name=tcmu-runner source packages
baseurl=https://1.chacra.ceph.com/r/tcmu-runner/master/9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/SRPMS
enabled=1
gpgcheck=0
type=rpm-md
ceph相關的ISCSI軟體包可以從redhat通過的源代碼進行編譯或者下載下傳centos已經編譯好的。
# yum install ceph-iscsi tcmu-runner targetcli python-rtslib -y
# ceph osd pool create rbd 150 150
# ceph osd pool application enable rbd rbd --yes-i-really-mean-it
2 建立配置檔案
建立/etc/ceph/iscsi-gateway.cfg ,此檔案主要設定iscsi服務的網關。
[config]
# Name of the Ceph storage cluster. A suitable Ceph configuration file allowing
# access to the Ceph storage cluster from the gateway node is required, if not
# colocated on an OSD node.
cluster_name = ceph
# Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph
# drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring
# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need to
# create cert/key files that are compatible for each iSCSI gateway node, that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
# to switch to https mode.
# To support the API, the bear minimum settings are:
api_secure = false
# Additional API configuration options are as follows, defaults shown.
# api_user = admin
# api_password = admin
# api_port = 5001
trusted_ip_list = 192.168.70.81,192.168.70.82,192.168.70.83
3、 同步檔案到其他節點
# scp /etc/ceph/iscsi-gateway.cfg node2:/etc/ceph
# scp /etc/ceph/iscsi-gateway.cfg node3:/etc/ceph
4、 啟動API服務
# systemctl daemon-reload
# systemctl enable rbd-target-api
# systemctl start rbd-target-api
# systemctl status rbd-target-api
● rbd-target-api.service - Ceph iscsi target configuration API
Loaded: loaded (/usr/lib/systemd/system/rbd-target-api.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2018-05-31 11:35:04 CST; 4s ago
Main PID: 25372 (rbd-target-api)
CGroup: /system.slice/rbd-target-api.service
└─25372 /usr/bin/python /usr/bin/rbd-target-api
May 31 11:35:04 node1 systemd[1]: Started Ceph iscsi target configuration API.
May 31 11:35:04 node1 systemd[1]: Starting Ceph iscsi target configuration API...
May 31 11:35:05 node1 rbd-target-api[25372]: Started the configuration object watcher
May 31 11:35:05 node1 rbd-target-api[25372]: Checking for config object changes every 1s
May 31 11:35:05 node1 rbd-target-api[25372]: * Running on http://0.0.0.0:5000/
5、配置ISCSI服務
5.1 建立target
# gwcli
/> cd iscsi-targets
/iscsi-target> create iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
/iscsi-targets> cd iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/
/iscsi-target...-gw:iscsi-igw> cd gateways
5.2 建立ISCSI網關
/iscsi-target...-igw/gateways> create node1 192.168.70.81
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...-igw/gateways> create node2 192.168.70.82
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...-igw/gateways> create node3 192.168.70.83
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...-igw/gateways> ls
o- gateways .................................................................................................. [Up: 3/3, Portals: 3]
o- node1 .................................................................................................. [192.168.70.81 (UP)]
o- node2 .................................................................................................. [192.168.70.82 (UP)]
o- node3 .................................................................................................. [192.168.70.83 (UP)]
如果作業系統非Centos或redhat,則需要加skipchecks=true參數。
/iscsi-target...-igw/gateways> create node1 192.168.70.81 skipchecks=true
遇到問題“The first gateway defined must be the local machine”,必須先建立本地節點
/iscsi-target...-igw/gateways> create node1 10.130.70.91
The first gateway defined must be the local machine
/iscsi-target...-igw/gateways> create node2 10.130.70.91
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
遇到問題Failed : Gateway creation failed, gateway(s) unavailable:node2(UNKNOWN state),需要三個節點都更新核心,更新核心參見:https://wuyeliang.blog.csdn.net/article/details/67644077
/iscsi-target...-igw/gateways> create node1 10.130.70.90 skipchecks=true
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
Failed : Gateway creation failed, gateway(s) unavailable:node2(UNKNOWN state)
遇到問題Is the API server running and in the right mode (http/https)?
注意關閉防火牆
systemctl stop firewalld
5.3 建立RBD image
> /iscsi-target...-igw/gateways> cd /disks
> /disks> create pool=rbd image=disk_1 size=90G
5.4 建立用戶端名稱
Linux平台可以檢視/etc/iscsi/initiatorname.iscsi檔案擷取InitiatorName。如果修改了預設的名稱,必須重新開機iscsid服務,否則在登入iscsi服務端的時候會報錯。
> /disks> cd /iscsi-target/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts
> /iscsi-target...eph-igw/hosts> create iqn.1994-05.com.redhat:rh7-client
5.5 設定用戶端認證
> /iscsi-target...at:rh7-client> auth username=myiscsiusername password=myiscsipassword
5.6 用戶端映射磁盤
/iscsi-target...at:rh7-client> disk add rbd/disk_1
五、windows的多路徑連接配接
下面分享下Windows下多路徑功能配置
如果沒有添加MultipathI/O功能,在iSCSI發起程式屬性——》裝置——》中看到“MPIO”顯示會是灰色。
安裝MPIO
配置MPIO支援ISCSI
連接配接ISCSI配置MPIO支援
然後選擇連接配接的存儲-選擇屬性配置另一條路徑
此時再檢視屬性
六.iSCSI initiator
- 安裝initiator與multipath工具
- iscsi-initiator-utils是通用initiator套件;
- device-mapper-multipath是多路徑工具
# yum install iscsi-initiator-utils device-mapper-multipath -y
- 設定multipath服務,啟用multipath服務,生成”/etc/multipath.conf”檔案
# mpathconf --enable --with_multipathd y
在”/etc/multipath.conf”檔案新增配置,針對LIO後端存儲設定多路徑ha
devices {
device {
vendor "LIO-ORG"
hardware_handler "1 alua"
path_grouping_policy "failover"
path_selector "queue-length 0"
failback 60
path_checker tur
prio alua
prio_args exclusive_pref_bit
fast_io_fail_tmo 25
no_path_retry queue
}
}
重新加載multinpath服務
# systemctl reload multipathd
-
iscsi discovery
1)設定chap認證
# 開啟initiator的chap認證,并設定username/password,與iscsi-target設定保持一緻;
# CHAP Settings部分,涉及57/61/62行
# vim /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = iscsiname
node.session.auth.password = iscsipassword
2)設定initiatoe-name
# 設定initiator-name,保持與iscsi-target設定的initiator-name一緻
# vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-04.192.168.70.50:iscsi-initiator
3)發現iscsi-target
# 發現iscsi存儲:iscsiadm -m discovery -t st -p ISCSI_IP,ISCSI_IP預設采用3260端口;
# 檢視iscsi發現記錄:iscsiadm -m node
# 删除iscsi發現記錄:iscsiadm -m node -o delete -T LUN_NAME -p ISCSI_IP
# iscsiadm -m discovery -t st -p 192.168.70.57
4)登陸iscsi-target
# 登入iscsi存儲:iscsiadm -m node -T LUN_NAME -p ISCSI_IP -l
# 登出iscsi存儲:iscsiadm -m node -T LUN_NAME -p ISCSI_IP -u
# 顯示會話情況:iscsiadm -m session
# iscsiadm -m node -T iqn.2018-09.192.168.70.5x:iscsi-gw -l
# 1個後端存儲,通過3條路徑連接配接
# multipath -ll
# 通過多路徑連接配接後端存儲,生成多個盤符;
# 通過multipath服務彙聚,生成盤符/dev/mapper/mpathx,mount時間直接使用;
# 或:lsscsi
# fdisk -l
- mount驗證
# mkfs.xfs /dev/mapper/mpatha
# 挂載分區
# mount /dev/mapper/mpatha /mnt
# 檢視挂載情況
# df -Th
- 修改/etc/fstab設定開機啟動挂載
/dev/mapper/mpatha1 /mnt xfs noatime,_netdev 0 0
參數說明:
# filesystem parameters列設定挂載時間;
# noatime:禁止更新檔案與目錄的inode通路時間,以獲得更快的通路速度;
# _netdev:辨別檔案系統位于網絡上,防止網絡啟動前挂載
# rbd
參考:
https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/
https://www.lagou.com/lgeduarticle/43742.html
https://blog.51cto.com/candon123/2125049
https://blog.51cto.com/ityunwei2017/1916244
https://www.cnblogs.com/netonline/p/10432653.html