一、軟體簡介
DRBD是一個用軟體實作的、無共享的、伺服器之間鏡像塊裝置内容的存儲複制解決方案。
MooseFS是一個具有容錯性的網絡分布式檔案系統。它把資料分散存放在多個實體伺服器上，而呈現給
使用者的則是一個統一的資源。
Moosefs：（mfs）存儲海量小檔案，支援FUSE。（國内企業使用比較多）
Pacemaker是一個叢集資料總管。它利用叢集基礎構件（corosync）提供的消息和成員管理能力來探
測并從節點或資源級别的故障中恢複，以實作群集服務的最大可用性。
尤為重要的是Pacemaker不是一個heartbeat的分支，似乎很多人存在這樣的誤解。Pacemaker是CRM項
目（亦名V2資料總管）的延續，該項目最初是為heartbeat而開發，但目前已經成為獨立項目。
Corosync是叢集管理套件的一部分，它在傳遞資訊的時候可以通過一個簡單的配置檔案來定義資訊傳遞
的方式和協定等。
CRM是一個指令行基于群集配置和管理工具。其目标是盡可能地協助基于pacemaker的高可用性叢集的
配置和維護。最為重要的是crm提供了互動界面，更加容易排錯。
二、需求
1、解決mfs-master單點故障
2、提供圖檔伺服器之類的分布式存儲，也可以在其他叢集中使用。
3、實作drbd的主備容災備份。
4、實作心跳檢測
5、實作服務（資源）的檢測及切換
6、實作高可用叢集
三、平台環境
OS：CentOS Linux release 7.3.1611 (Core)
kernel:3.10.0-514.el7.x86_64
網絡資訊規劃如下表：
| 名稱 | hostname | IP |
| VIP | 飄移 | 192.168.40.200 |
| mfs-master1(drbd) | node4 | 192.168.40.131 |
| mfs-master2(drbd) | node5 | 192.168.40.132 |
| mfs-metalog server | node6 | 192.168.40.133 |
| mfs-chunk server1 | node7 | 192.168.40.134 |
| mfs-chunk server2 | node8 | 192.168.40.134 |
| client | node1 | 192.168.40.128 |
網絡拓撲如下：
四、環境準備
1、 修改hosts檔案保證hosts之間能夠互相通路。
[[email protected] ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.40.131 node4
192.168.40.132 node5
192.168.40.133 node6
192.168.40.134 node7
192.168.40.135 node8
2、同步時間
ntpdate cn.pool.ntp.org
3、mfs的各個server之間實作ssh互信（共5台server），指令如下：
ssh-keygen
ssh-copy-id node#主機名或者IP
五、安裝軟體
1、安裝drbd
在node4和node5先劃分出一個分區，而且不格式化那麼快。如下：
[[email protected] ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa56b82b0
Device Boot Start End Blocks Id System
/dev/sdb1 2048 10485759 5241856 83 Linux
[[email protected] ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x066827f3
Device Boot Start End Blocks Id System
/dev/sdb1 2048 10485759 5241856 83 Linux
安裝derb并安裝drbd
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum install -y kmod-drbd84 drbd84-utils
配置drbd全局配置檔案
[r[email protected] mfs]# cat /etc/drbd.d/global_common.conf
# DRBD is the result of over a decade of development by LINBIT.
# In case you need professional services for DRBD or have
# feature requests visit http://www.linbit.com
global {
usage-count no;
# Decide what kind of udev symlinks you want for "implicit" volumes
# (those without explicit volume <vnr> {} block, implied vnr=0):
# /dev/drbd/by-resource/<resource>/<vnr> (explicit volumes)
# /dev/drbd/by-resource/<resource> (default for implict)
udev-always-use-vnr; # treat implicit the same as explicit volumes
# minor-count dialog-refresh disable-ip-verification
# cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600;
}
common {
protocol C;
handlers {
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when choosing your poison.
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
# quorum-lost "/usr/lib/drbd/notify-quorum-lost.sh root";
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
}
options {
# cpu-mask on-no-data-accessible
# RECOMMENDED for three or more storage nodes with DRBD 9:
# quorum majority;
# on-no-quorum suspend-io | io-error;
}
disk {
on-io-error detach;
# size on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
}
net {
# protocol timeout max-epoch-size max-buffers
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
}
syncer {
rate 1024M;
}
}
/etc/drbd.conf #主配置檔案
/etc/drbd.d/global_common.conf #全局配置檔案
[[email protected] mfs]# cat /etc/drbd.d/mfs.res
resource mfs{
protocol C;
meta-disk internal;
device /dev/drbd1;
syncer{
verify-alg sha1;
}
net{
allow-two-primaries;
}
on node4 {
disk /dev/sdb1;
address 192.168.40.131:7789;
}
on node5 {
disk /dev/sdb1;
address 192.168.40.132:7789;
}
}
把配置檔案 copy 到對面的機器上：
[[email protected] ~]# scp -rp /etc/drbd.d/* node5:/etc/drbd.d/
global_common.conf 100% 2618 2.6KB/s 00:00
mfs.res 100% 248 0.2KB/s 00:00
在 node4 上面啟動：（注意：我在此處遇到報錯，如果你也遇到錯誤或者其他配置遇錯也一樣 ，
那就請看文章末尾的報錯連結，謝謝。）
[[email protected] ~]# drbdadm create-md mfs
initializing activity log
initializing bitmap (160 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.
檢視核心是否已經加載了子產品：
[[email protected] ~]# modprobe drbd
[[email protected] ~]# lsmod | grep drbd
drbd 396875 0
libcrc32c 12644 4 xfs,drbd,nf_nat,nf_conntrack
啟動mfs資源：
[[email protected] ~]# drbdadm up mfs
[[email protected] ~]# drbdadm --force primary mfs
[[email protected] ~]# cat /proc/drbd
version: 8.4.10-1 (api:1/proto:86-101)
GIT-hash: a4d5de01fffd7e4cde48a080e2c686f9e8cebf4c build by [email protected], 2017-09-15 14:23:22
1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s
ns:0 nr:0 dw:0 dr:912 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:5241660
在node5執行
[[email protected] ~]# drbdadm create-md mfs
initializing activity log
initializing bitmap (160 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.
[[email protected] ~]# modprobe drbd
[[email protected] ~]# drbdadm up mfs
然後 可以檢視資料同步的狀态：（需要等待一段時間）
[[email protected] ~]# cat /proc/drbd
version: 8.4.10-1 (api:1/proto:86-101)
GIT-hash: a4d5de01fffd7e4cde48a080e2c686f9e8cebf4c build by [email protected], 2017-09-15 14:23:22
1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
ns:0 nr:151552 dw:151552 dr:0 al:8 bm:0 lo:1 pe:3 ua:0 ap:0 ep:1 wo:f oos:5090108
[>....................] sync'ed: 3.0% (4968/5116)M
finish: 0:03:54 speed: 21,648 (21,648) want: 36,640 K/sec
格式化并挂載看看能否正常使用
[[email protected] ~]# mkfs.xfs -f /dev/drbd1
meta-data=/dev/drbd1 isize=512 agcount=4, agsize=327604 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=1310415, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[[email protected] ~]# mount /dev/drbd1 /mnt
[[email protected] ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl-root 17G 2.1G 15G 13% /
devtmpfs 478M 0 478M 0% /dev
tmpfs 489M 0 489M 0% /dev/shm
tmpfs 489M 6.6M 482M 2% /run
tmpfs 489M 0 489M 0% /sys/fs/cgroup
/dev/sda1 1014M 167M 848M 17% /boot
tmpfs 98M 0 98M 0% /run/user/0
/dev/drbd1 5.0G 33M 5.0G 1% /mnt
這就證明drbd可以正常使用了，取消挂載
[[email protected] ~]# umount /mnt
2、編譯安裝MFS
這次才使用的是moosefs,目前是國内的主流。
這次測試的版本moosefs-3.0.96。
安裝依賴包
yum install zlib-devel gcc -y
上傳moosefs-3.0.96，也就是v3.0.96.tar.gz，或者去GitHub下載下傳，
wgethttps://github.com/moosefs/moosefs/archive/v3.0.96.tar.gz
後續将用到crmsh
[[email protected] src]# ls
crmsh-2.3.2.tar v3.0.96.tar.gz
[[email protected] src]# scp v3.0.96.tar.gz node5:/usr/local/src/
v3.0.96.tar.gz 100% 1092KB 1.1MB/s 00:00
[[email protected] src]# scp v3.0.96.tar.gz node6:/usr/local/src/
v3.0.96.tar.gz 100% 1092KB 1.1MB/s 00:00
[[email protected] src]# scp v3.0.96.tar.gz node7:/usr/local/src/
v3.0.96.tar.gz 100% 1092KB 1.1MB/s 00:00
[[email protected] src]# scp v3.0.96.tar.gz node8:/usr/local/src/
v3.0.96.tar.gz 100% 1092KB 1.1MB/s 00:00
[[email protected] src]# scp v3.0.96.tar.gz node1:/usr/local/src/
建立mfs使用者，切記各個mfs-server的mfs的UID必須一緻。否者會失敗
[[email protected] src]# useradd -u 1000 mfs
[[email protected] src]# useradd -u 1000 mfs
[[email protected] src]# useradd -u 1000 mfs
[[email protected] src]# useradd -u 1000 mfs
[[email protected] src]# useradd -u 1000 mfs
挂載/dev/drbd1到/usr/local/mfs（ 隻需要在一台mfsmaster上安裝即可）
[[email protected] src]# drbd-overview
NOTE: drbd-overview will be deprecated soon.
Please consider using drbdtop.
1:mfs/0 Connected Primary/Secondary UpToDate/UpToDate
[[email protected] src]# mkdir /usr/local/mfs
[[email protected] src]# chown -R mfs:mfs /usr/local/mfs
[[email protected] src]# mount /dev/drbd1 /usr/local/mfs
開始編譯 moosefs-3.0.96
[[email protected] src]# tar -xf v3.0.96.tar.gz
[[email protected] src]# cd /usr/local/src/moosefs-3.0.96/
[[email protected] moosefs-3.0.96]# ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --disable-mfschunkserver --disable-mfsmount
由于node4是mfs-master，是以不需要裝載--disable-mfschunkserver、--disable-mfsmount
[[email protected] moosefs-3.0.96]# make && make install
配置master：
[[email protected] mfs]# pwd
/usr/local/mfs/etc/mfs
[[email protected] mfs]# ls
mfsexports.cfg.sample mfsmaster.cfg.sample mfsmetalogger.cfg.sample mfstopology.cfg.sample
[[email protected] mfs]# cp mfsexports.cfg.sample mfsexports.cfg
[[email protected] mfs]# cp mfsmaster.cfg.sample mfsmaster.cfg
對于mfsmaster.cfg檔案， 因為是官方的，預設配置，我們投入即可使用。
需要修改的檔案是mfsexports.cfg添加挂載目錄的權限及密碼（隻要在最後添加）
vim mfsexports.cfg
* / rw,alldirs,mapall=mfs:mfs,password=aizhen
* . rw
開啟中繼資料檔案預設是empty檔案，需要我們手工打開：
[[email protected] mfs]# cp /usr/local/mfs/var/mfs/metadata.mfs.empty /usr/local/mfs/var/mfs/metadata.mfs
啟動master：
[[email protected] mfs]# /usr/local/mfs/sbin/mfsmaster start
open files limit has been set to: 16384
working directory: /usr/local/mfs/var/mfs
lockfile created and locked
initializing mfsmaster modules ...
exports file has been loaded
mfstopology configuration file (/usr/local/mfs/etc/mfstopology.cfg) not found - using defaults
loading metadata ...
metadata file has been loaded
no charts data file - initializing empty charts
master <-> metaloggers module: listen on *:9419
master <-> chunkservers module: listen on *:9420
main master server module: listen on *:9421
mfsmaster daemon initialized properly
現在編寫mfsmaster的啟動腳本，才用systemctl啟動。
[[email protected] mfs]# cat /usr/lib/systemd/system/mfsmaster.service
[Unit]
Description=mfs
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/mfs/sbin/mfsmaster start
ExecStop=/usr/local/mfs/sbin/mfsmaster stop
PrivateTmp=true
[Install]
WantedBy=multi-user.target
[[email protected] mfs]# chmod 775 /usr/lib/systemd/system/mfsmaster.service
測試mfsmaster.service腳本能否正常使用
[[email protected] mfs]# systemctl start mfsmaster.service
[[email protected] mfs]# systemctl status mfsmaster.service
● mfsmaster.service - mfs
Loaded: loaded (/usr/lib/systemd/system/mfsmaster.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2017-10-28 13:04:40 EDT; 7s ago
Process: 7555 ExecStart=/usr/local/mfs/sbin/mfsmaster start (code=exited, status=0/SUCCESS)
Main PID: 7557 (mfsmaster)
CGroup: /system.slice/mfsmaster.service
└─7557 /usr/local/mfs/sbin/mfsmaster start
[[email protected] mfs]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9419 0.0.0.0:* LISTEN 7557/mfsmaster
tcp 0 0 0.0.0.0:9420 0.0.0.0:* LISTEN 7557/mfsmaster
tcp 0 0 0.0.0.0:9421 0.0.0.0:* LISTEN 7557/mfsmaster
設定mfsmaster開機自啟動：
[[email protected] mfs]# systemctl enable mfsmaster
Created symlink from /etc/systemd/system/multi-user.target.wants/mfsmaster.service to /usr/lib/systemd/system/mfsmaster.service.
将mfsmaster.service複制到node5
[[email protected] mfs]# scp /usr/lib/systemd/system/mfsmaster.service node5:/usr/lib/systemd/system/
mfsmaster.service 100% 217 0.2KB/s 00:00
在node5也設定mfsmaster開機自啟動；
[[email protected] ~]# systemctl enable mfsmaster
Created symlink from /etc/systemd/system/multi-user.target.wants/mfsmaster.service to /usr/lib/systemd/system/mfsmaster.service.
在node5建立必須的目錄：
[[email protected] ~]# mkdir /usr/local/mfs
[[email protected] ~]# chown -R mfs:mfs /usr/local/mfs
現在開始測試drbd能否主從正常切換，并且實作mfsmaster的切換（切換到node5）
在node4上
[[email protected] mfs]# systemctl stop mfsmaster
[[email protected] mfs]# cd
[[email protected] ~]# umount /usr/local/mfs
[[email protected] ~]# drbdadm secondary mfs
在node5上
[[email protected] ~]# mkdir /usr/local/mfs
[[email protected] ~]# chown -R mfs:mfs /usr/local/mfs
[[email protected] ~]# drbdadm primary mfs
[[email protected] ~]# mount /dev/drbd1 /usr/local/mfs
[[email protected] ~]# systemctl start mfsmaster
[[email protected] ~]# systemctl status mfsmaster
● mfsmaster.service - mfs
Loaded: loaded (/usr/lib/systemd/system/mfsmaster.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2017-10-28 13:12:01 EDT; 13s ago
Process: 1646 ExecStart=/usr/local/mfs/sbin/mfsmaster start (code=exited, status=0/SUCCESS)
Main PID: 1648 (mfsmaster)
CGroup: /system.slice/mfsmaster.service
└─1648 /usr/local/mfs/sbin/mfsmaster start
[[email protected] ~]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9419 0.0.0.0:* LISTEN 1648/mfsmaster
tcp 0 0 0.0.0.0:9420 0.0.0.0:* LISTEN 1648/mfsmaster
以上說明測試沒問題！！！
在node6安裝metalogger server
啰嗦一下：Metalogger Server 是 Master Server 的備份伺服器。是以，Metalogger Server 的安裝步驟和 Master Server 的安裝步驟相同。并且，最好使用和 Master Server 配置一樣的伺服器來做 Metalogger Server。這樣，一旦主伺服器master當機失效，我們隻要導入備份資訊changelogs到中繼資料檔案，備份伺服器可直接接替故障的master繼續提供服務。可以根據成本來決定是否需要添加這台伺服器。
編譯安裝metalogger server
[[email protected] src]# tar -xf v3.0.96.tar.gz
[[email protected] src]# cd moosefs-3.0.96/
[[email protected] moosefs-3.0.96]# ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --disable-mfschunkserver --disable-mfsmount
[[email protected] moosefs-3.0.96]# make && make install
配置metalogger server
[[email protected] moosefs-3.0.96]# cd /usr/local/mfs/etc/mfs/
[[email protected] mfs]# ls
mfsexports.cfg.sample mfsmaster.cfg.sample mfsmetalogger.cfg.sample mfstopology.cfg.sample
[[email protected] mfs]# cp mfsmetalogger.cfg.sample mfsmetalogger.cfg
[[email protected] mfs]# vim mfsmetalogger.cfg
MASTER_HOST = 192.168.40.200
将其指向VIP 或者先指向某台mfsmaster做測試
啟動metalogger server：（啟動沒問題就寫啟動腳本，并設定開機自啟動）
[[email protected] ~]# /usr/local/mfs/sbin/mfsmetalogger start
open files limit has been set to: 4096
working directory: /usr/local/mfs/var/mfs
lockfile created and locked
initializing mfsmetalogger modules ...
mfsmetalogger daemon initialized properly
[[email protected] ~]# mv /usr/lib/systemd/system/mfsmaster.service /usr/lib/systemd/system/mfsmetalog.service
[[email protected] ~]# cat /usr/lib/systemd/system/mfsmetalog.service
[Unit]
Description=mfs
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/mfs/sbin/mfsmetalogger start
ExecStop=/usr/local/mfs/sbin/mfsmetalogger stop
PrivateTmp=true
[Install]
WantedBy=multi-user.target
[[email protected] ~]# /usr/local/mfs/sbin/mfsmetalogger stop
sending SIGTERM to lock owner (pid:2093)
waiting for termination terminated
[[email protected] ~]# systemctl start mfsmetalog
[[email protected] ~]# systemctl status mfsmetalog
● mfsmetalog.service - mfs
Loaded: loaded (/usr/lib/systemd/system/mfsmetalog.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2017-10-28 20:26:28 EDT; 12s ago
Process: 2113 ExecStart=/usr/local/mfs/sbin/mfsmetalogger start (code=exited, status=0/SUCCESS)
Main PID: 2115 (mfsmetalogger)
CGroup: /system.slice/mfsmetalog.service
└─2115 /usr/local/mfs/sbin/mfsmetalogger start
[[email protected] ~]# systemctl enable mfsmetalog
Created symlink from /etc/systemd/system/multi-user.target.wants/mfsmetalog.service to /usr/lib/systemd/system/mfsmetalog.service.
在node7、node8編譯安裝chunk server。
node7和node8配置一樣，以node7為例：
[[email protected] src]# tar -xf v3.0.96.tar.gz
[[email protected] src]# cd moosefs-3.0.96/
[[email protected] moosefs-3.0.96]# ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --disable-mfsmaster --disable-mfsmount
[[email protected] moosefs-3.0.96]# make && make install
配置mfschunkserver.cfg的檔案
[[email protected] moosefs-3.0.96]# cd /usr/local/mfs/etc/mfs/
[[email protected] mfs]# ls
mfschunkserver.cfg.sample mfshdd.cfg.sample mfsmetalogger.cfg.sample
[[email protected] mfs]# cp mfschunkserver.cfg.sample mfschunkserver.cfg
[[email protected] mfs]# cp mfshdd.cfg.sample mfshdd.cfg
[[email protected] mfs]# vim mfschunkserver.cfg
MASTER_HOST = 192.168.40.200
将其指向VIP 或者先指向某台mfsmaster做測試
配置mfshdd.cfg的檔案
mfshdd.cfg該檔案用來設定你将 Chunk Server 的哪個目錄共享出去給 Master Server進行管理。當然，雖然這裡填寫的是共享的目錄，但是這個目錄後面最好是一個單獨的分區。
[[email protected] mfs]# mkdir /mfsdata
[[email protected] mfs]# chown -R mfs:mfs /mfsdata
[[email protected] mfs]# vim mfshdd.cfg
直接在最後一行加
/mfsdata
啟動chunkserver：（啟動沒問題就寫啟動腳本，并設定開機自啟動）
[[email protected] mfs]# /usr/local/mfs/sbin/mfschunkserver start
[[email protected] mfs]# /usr/local/mfs/sbin/mfschunkserver stop
[[email protected] mfs]# cat /usr/lib/systemd/system/mfschunk.service
[Unit]
Description=mfs
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/mfs/sbin/mfschunkserver start
ExecStop=/usr/local/mfs/sbin/mfschunkserver stop
PrivateTmp=true
[Install]
WantedBy=multi-user.target
[[email protected] mfs]# systemctl enable mfschunk
MFS的服務已經配置完成。
先停止node4、node5的服務
[[email protected] ~]# systemctl stop mfsmaster
[[email protected] ~]# systemctl stop drbd
[[email protected] ~]# systemctl stop mfsmaster
[[email protected] ~]# systemctl stop drbd
3、安裝corosync和pacemaker
在node4、node5解決依賴問題
[[email protected] ~]# yum install -y pacemaker pcs psmisc policycoreutils-python
生命周期管理工具：
Pcs:agent(pcsd)
Crash:pssh
開啟pcs并設定開機自啟動：
[[email protected] ~]# systemctl start pcsd
[[email protected] ~]# systemctl enable pcsd
修改hacluster使用者的密碼：
[[email protected] ~]# echo 000000 | passwd --stdin hacluster
[[email protected] ~]# echo 000000 | passwd --stdin hacluster
注冊pcs叢集主機（預設使用者是hacluster和密碼）{隻要在一台node執行}
[[email protected] ~]# pcs cluster auth node4 node5
Username: hacluster
Password:
node5: Authorized
node4: Authorized
在叢集上注冊兩台叢集：
[[email protected] ~]# pcs cluster setup --name mysqlcluster node4 node5 --force
Destroying cluster on nodes: node4, node5...
node5: Stopping Cluster (pacemaker)...
node4: Stopping Cluster (pacemaker)...
node5: Successfully destroyed cluster
node4: Successfully destroyed cluster
Sending 'pacemaker_remote authkey' to 'node4', 'node5'
node5: successful distribution of the file 'pacemaker_remote authkey'
node4: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
node4: Succeeded
node5: Succeeded
Synchronizing pcsd certificates on nodes node4, node5...
node5: Success
node4: Success
Restarting pcsd on the nodes in order to reload the certificates...
node5: Success
node4: Success
我們看到生成來corosync.conf配置檔案：
[[email protected] ~]# ll /etc/corosync/
total 16
-rw-r--r-- 1 root root 385 Oct 28 21:37 corosync.conf
-rw-r--r-- 1 root root 2881 Sep 6 12:53 corosync.conf.example
-rw-r--r-- 1 root root 767 Sep 6 12:53 corosync.conf.example.udpu
-rw-r--r-- 1 root root 3278 Sep 6 12:53 corosync.xml.example
drwxr-xr-x 2 root root 6 Sep 6 12:53 uidgid.d
啟動叢集：
[[email protected] ~]# pcs cluster start --all
node4: Starting Cluster...
node5: Starting Cluster...
## 相當于啟動來pacemaker和corosync
在node4、node5将pacemaker和corosync設定為開機自啟動：
[[email protected] ~]# systemctl enable pacemaker
[[email protected] ~]# systemctl enable corosync
## 因為我們沒有配置STONITH裝置，是以我們下面要關閉：
[[email protected] ~]# pcs property set stonith-enabled=false
叢集我們可以下載下傳安裝crmsh來操作(從github來下載下傳，然後解壓直接安裝)：隻在一個節點安裝即可。
主要是crmsh使用互動界面，友善排錯。
編譯安裝crmsh-2.3.2：
[[email protected] src]# tar -xf crmsh-2.3.2.tar
[[email protected] src]# cd crmsh-2.3.2
[[email protected] crmsh-2.3.2]# python setup.py install
哈哈，就是喜歡這麼和諧的界面：
[[email protected] ~]# crm
crm(live)# status
Stack: corosync
Current DC: node4 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Sat Oct 28 21:50:11 2017
Last change: Sat Oct 28 21:45:22 2017 by root via cibadmin on node5
2 nodes configured
0 resources configured
Online: [ node4 node5 ]
No resources
crm(live)#
下面開始配置資源：
crm(live)configure# primitive mfs_drbd ocf:linbit:drbd params drbd_resource=mfs op monitor role=Master interval=10 timeout=20 op monitor role=Slave interval=20 timeout=20 op start timeout=240 op stop timeout=100
crm(live)configure# verify
crm(live)configure#
crm(live)configure# ms ms_mfs_drbd mfs_drbd meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
crm(live)configure# verify
crm(live)configure#
确定verify沒問題即可（若有問題可以直接使用edit修改檔案）
crm(live)configure# commit
配置挂載資源：
crm(live)configure# primitive mfsstore ocf:heartbeat:Filesystem params device=/dev/drbd1 directory=/usr/local/mfs fstype=xfs op start timeout=60 op stop timeout=60
crm(live)configure# verify
crm(live)configure# colocation ms_mfs_drbd_with_mfsstore inf: mfsstore ms_mfs_drbd
crm(live)configure# order ms_mfs_drbd_before_mystore Mandatory: ms_mfs_drbd:promote mfsstore:start
crm(live)configure# verify
crm(live)configure# commit
colocation是綁定親緣關系， order是決定那個服務先啟動（很重要）
配置mfs資源：
crm(live)configure# primitive mfs systemd:mfsmaster op monitor timeout=100 interval=30 op start timeout=100 interval=0 op stop timeout=100 interval=0
crm(live)configure# verify
crm(live)configure#
crm(live)configure# colocation mfs_with_mystore inf: mfs mfsstore
crm(live)configure# order mfsstore_befor_mfs Mandatory: mfsstore mfs
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure#
配置VIP資源：
crm(live)configure# primitive vip ocf:heartbeat:IPaddr params ip=192.168.40.200
crm(live)configure# colocation vip_with_msf inf: vip mfs
crm(live)configure# verify
crm(live)configure# commit
使用show檢視配置的内容：
crm(live)configure# show
node 1: node4
node 2: node5
primitive mfs systemd:mfsmaster \
op monitor timeout=100 interval=30 \
op start timeout=100 interval=0 \
op stop timeout=100 interval=0
primitive mfs_drbd ocf:linbit:drbd \
params drbd_resource=mfs \
op monitor role=Master interval=10 timeout=20 \
op monitor role=Slave interval=20 timeout=20 \
op start timeout=240 interval=0 \
op stop timeout=100 interval=0
primitive mfsstore Filesystem \
params device="/dev/drbd1" directory="/usr/local/mfs" fstype=xfs \
op start timeout=60 interval=0 \
op stop timeout=60 interval=0
primitive vip IPaddr \
params ip=192.168.40.200
ms ms_mfs_drbd mfs_drbd \
meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
colocation mfs_with_mystore inf: mfs mfsstore
order mfsstore_befor_mfs Mandatory: mfsstore mfs
order ms_mfs_drbd_before_mystore Mandatory: ms_mfs_drbd:promote mfsstore:start
colocation ms_mfs_drbd_with_mfsstore inf: mfsstore ms_mfs_drbd
colocation vip_with_msf inf: vip mfs
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.16-12.el7_4.4-94ff4df \
cluster-infrastructure=corosync \
cluster-name=mfscluster \
stonith-enabled=false
所有服務端的所有配置已經配完！！！
檢視各項資源已經正常使用
crm(live)# status
Stack: corosync
Current DC: node4 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Sat Oct 28 22:21:47 2017
Last change: Sat Oct 28 22:17:18 2017 by root via cibadmin on node4
2 nodes configured
5 resources configured
Online: [ node4 node5 ]
Full list of resources:
Master/Slave Set: ms_mfs_drbd [mfs_drbd]
Masters: [ node5 ]
Slaves: [ node4 ]
mfsstore (ocf::heartbeat:Filesystem): Started node5
mfs (systemd:mfsmaster): Started node5
vip (ocf::heartbeat:IPaddr): Started node5
mfs伺服器已經做好提供挂載服務。
下面開始配置用戶端（也就是需要挂載分布式存儲的後端伺服器，例如：圖檔伺服器之類等）
在node1解決依賴問題。（等待時間有點久）
[[email protected] etc]# yum install fuse fuse-devel zlib-devel gcc -y
編譯安裝moosefs-3.0.96
[[email protected] src]# tar -xf v3.0.96.tar.gz
[[email protected] src]# cd moosefs-3.0.96/
[[email protected] moosefs-3.0.96]# ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --disable-mfsmaster --disable-mfschunkserver --enable-mfsmount
[[email protected] moosefs-3.0.96]# make && make install
[[email protected] moosefs-3.0.96]# mkdir /mfsdata
[[email protected] moosefs-3.0.96]# chown -R mfs:mfs /mfsdata
測試挂載：
[[email protected] moosefs-3.0.96]# /usr/local/mfs/bin/mfsmount /mfsdata -H 192.168.40.200 -p
MFS Password:
mfsmaster accepted connection with parameters: read-write,restricted_ip,map_all ; root mapped to nginx:nginx ; users mapped to nginx:nginx
[[email protected] moosefs-3.0.96]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl-root 17G 9.0G 8.1G 53% /
devtmpfs 478M 0 478M 0% /dev
tmpfs 489M 0 489M 0% /dev/shm
tmpfs 489M 6.6M 482M 2% /run
tmpfs 489M 0 489M 0% /sys/fs/cgroup
/dev/sda1 1014M 138M 877M 14% /boot
tmpfs 98M 0 98M 0% /run/user/0
192.168.40.200:9421 34G 4.5G 30G 14% /mfsdata
密碼是配置mfsmaster伺服器的mfsexport.cfg配置的密碼（我配置的是aizhen）
測試是否能正常寫入資料：
[[email protected] moosefs-3.0.96]# cd /mfsdata/
[[email protected] mfsdata]# ls
[[email protected] mfsdata]# touch a.txt
[[email protected] mfsdata]# echo 123 >> a.txt
[[email protected] mfsdata]# cat a.txt
123
至此，所有的配置已經完成。
六、注意事項及報錯。
本次測試的所有注意事項及錯誤都在我的另外一篇博文，
主要是為了友善自己的排錯。請諒解，
錯誤連結：點選打開連結
本文為部落客原創，轉載請注明本文的出處，謝謝