環境:
ceph L版本12.2.1更新到12.2.12
這個問題是由于更新後進行12.2.12環境中的使用ceph-disk 進行osd部署時出現如下問題,執行指令
ceph-disk -v prepare /dev/sdb;ceph-disk -v activate /dev/sdb1
出現如下問題,出現這個問題之前我的磁盤用作部署過12.2.1版本的osd,但是我已經使用
sgdisk -z -og /dev/sdb
格式化過磁盤
command_check_call: Running command: /usr/bin/ceph-osd --cluster ceph --mkfs -i 2 --monmap /var/lib/ceph/tmp/mnt.uYoUWC/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.uYoUWC --osd-uuid e6ac88e7-7929-4405-937a-8686a30a0037 --setuser ceph --setgroup ceph
2019-11-03 00:32:35.588340 7fcc21bddd00 -1 bluestore(/var/lib/ceph/tmp/mnt.uYoUWC/block) _check_or_set_bdev_label bdev /var/lib/ceph/tmp/mnt.uYoUWC/block fsid 0fb18eac-7c3b-47a8-9b1f-64407ccf1905 does not match our fsid e6ac88e7-7929-4405-937a-8686a30a0037
2019-11-03 00:32:35.844858 7fcc21bddd00 -1 bluestore(/var/lib/ceph/tmp/mnt.uYoUWC) mkfs fsck found fatal error: (5) Input/output error
2019-11-03 00:32:35.844890 7fcc21bddd00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
2019-11-03 00:32:35.845005 7fcc21bddd00 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.uYoUWC: (5) Input/output error
根據官網bug跟蹤的解決過程unable to create bluestore osd using ceph-disk 也進行了一些測試,發現即使時我使用
sgdisk
格式化了磁盤分區之後,仍然可以在磁盤的前1個G容量中看到有12.2.1相關資料。即我在使用12.2.1版本部署過osd的磁盤去部署目前版本12.2.12時,上一個版本的一些fsid,uuid,osd編号等一些rockdb中繼資料資訊仍然存在于目前磁盤的扇區中,這個時候我即使格式化磁盤仍然無法清楚掉這一些資料得,檢視sgdisk源碼,使用-o參數清楚磁盤分區資料隻會清除GPT 格式的資料,并不會清除MBR格式資料。
對已經
sgdisk -z -og /dev/sdk
之後的磁盤執行如下操作:
[root@node1 zhg]# dd if=/dev/zero of=/dev/sdk bs=1M count=1G
^C4647+0 records in
4647+0 records out
4872732672 bytes (4.9 GB) copied, 21.5581 s, 226 MB/s
[root@node1 zhg]# dd if=/dev/sdk of=mbr.txt bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 9.53945 s, 113 MB/s
檢視dd出來的1G容量的資料
0010c000 49 4e 41 ed 02 01 00 00 00 00 00 a7 00 00 00 a7 |INA.............|
0010c010 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 02 |................|
0010c020 00 00 00 00 00 00 00 00 5d bd af 16 04 b6 73 f2 |........].....s.|
0010c030 5d bd af 16 04 b6 73 f2 00 00 00 00 00 00 00 86 |].....s.........|
0010c040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
0010c050 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 |................|
0010c060 ff ff ff ff 09 00 00 00 00 18 09 00 50 63 65 70 |............Pcep|
0010c070 68 5f 66 73 69 64 00 00 00 1b 04 00 88 66 73 69 |h_fsid.......fsi|
0010c080 64 00 00 00 1c 05 00 b8 6d 61 67 69 63 00 00 00 |d.......magic...|
0010c090 1d 0a 00 f0 62 6c 6f 63 6b 5f 75 75 69 64 00 00 |....block_uuid..|
0010c0a0 00 1e 05 01 08 62 6c 6f 63 6b 00 00 00 1f 04 01 |.....block......|
0010c0b0 38 74 79 70 65 00 00 00 20 07 01 48 6b 65 79 72 |8type... ..Hkeyr|
0010c0c0 69 6e 67 00 00 00 21 06 01 80 77 68 6f 61 6d 69 |ing...!...whoami|
0010c0d0 00 00 00 22 0f 01 98 61 63 74 69 76 61 74 65 2e |..."...activate.|
0010c0e0 6d 6f 6e 6d 61 70 00 00 00 23 00 00 00 00 00 00 |monmap...#......|
0010c0f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
可以看到,雖然已經格式化磁盤了,但是扇區中仍然存在上一個版本osd相關中繼資料資訊。
ceph osd rm osd.id #删除添加失敗的osd
ceph auth rm osd.id #删除添加失敗的osd的密鑰
dd if=/dev/zero of=/dev/sdk bs=1M count=1024 #使用0重寫前一個G内容
sgdisk -Z -g /dev/sdk #先破壞磁盤所有分區,再将分區由mgr轉為gpt格式分區
ceph-disk -v prepare /dev/sdk
ceph-disk -v activate /dev/sdk1
[root@node1 zhg]# dd if=/dev/sdk of=mbr.txt bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 9.53945 s, 113 MB/s
[root@node1 zhg]# hexdump -C mbr.txt
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|