ceph-objectstore-tool工具使用详解

文章目录

• ​​简介​​
• ​​使用​​

• ​​OSD相关操作​​
• ​​PG相关操作​​
• ​​对象相关操作​​

• ​​总结​​

简介

​

​ceph-objectstore-tool​

​​工具，能够操作到ceph最底层的数据，包括pg,对象层级。它能够对底层pg以及对象相关数据进行获取、修改。并能够对一些问题pg和对象进行简单修复。所以使用该工具进行操作的时候需要谨慎（涉及到修改的操作最好备份一份数据），防止误操作造成数据丢失。

该工具的使用主要针对​​

​对象​

​​和​

​PG​

​,所以这里我们的使用主要为以下两种

• ​

  ​PG​

  ​的相关操作

info #查看pg的信息
log  #查看pg的元数据信息
remove #从当前osd移除pg
mkfs #重新初始化osd
fsck #检查bluefs数据正确性
export #导出pg元数据
import #导入pg元数据
list #列出pg内部对象或者当前osd所有对象
fix-lost #修复pg丢失的对象
list-pgs #列出当前osd所有pg
rm-past-intervals 
dump-journal #针对filestore，打印日志
dump-super #打印osd的超级块信息
meta-list #打印元数据信息列表
get-osdmap #获取osdmap
set-osdmap #设置osd map
get-inc-osdmap #从当前osd 获取inc 信息
set-inc-osdmap #将inc设置进入当前osd信息
mark-complete #标记为complete，让bluestore认为可以pg可以选举出权威日志      

• ​

  ​对象​

  ​的相关操作,这里主要列出常用的对象操作

list-attrs  #列出对象的一些属性
 list-omap #列出对象的omp信息
 remove|removeall #移除对象或者移除所有对象
 dump #打印对象元数据信息      

使用

基本命令使用如下：

​​

​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore xxx​

​​一般前半部分如左，指定osd路径，指定存储引擎（​

​bluestore​

​​或者​

​filestore​

​）

OSD相关操作

• 检查bluestore的bluefs是否文件系统被损坏，并尝试修复

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op fsck​

  ​

• 列出当前osd所有对象

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op list​

  ​

• 获取当前osd的osdmap信息，并指定对应的输出文件

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op get-osdmap --file 1.txt​

  ​

[root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op get-osdmap --file 1.txt
osdmap#1457 exported.      

• 导入一个osdmap信息到当前osd

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op set-osdmap --file 1.txt​

  ​

[root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op set-osdmap --file 1.txt
Wrote osdmap.145      

• 获取当前osd的超级块信息

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op dump-super​

  ​ 输出如下：

{
"cluster_fsid": "fa27f041-0ce9-4df1-a4bd-5e37678834bd",
"osd_fsid": "c03c2fdb-cfd4-42bd-8e32-61e359281078",
"whoami": 8,
"current_epoch": 1457,
"oldest_map": 784,
"newest_map": 1457,
"weight": 0.000000,
"compat": {
    "compat": {},
    "ro_compat": {},
    "incompat": {
        "feature_1": "initial feature set(~v.18)",
        "feature_2": "pginfo object",
        "feature_3": "object locator",
        "feature_4": "last_epoch_clean",
        "feature_5": "categories",
        "feature_6": "hobjectpool",
        "feature_7": "biginfo",
        "feature_8": "leveldbinfo",
        "feature_9": "leveldblog",
        "feature_10": "snapmapper",
        "feature_11": "sharded objects",
        "feature_12": "transaction hints",
        "feature_13": "pg meta object",
        "feature_14": "explicit missing set",
        "feature_15": "fastinfo pg attr",
        "feature_16": "deletes in missing set"
    }
},
"clean_thru": 1457,
"last_epoch_mounted": 1456
}      

• 获取当前osd的所有pg

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --op list-pgs​

  ​ 其中​

  ​--op​

  ​表示对当前osd或者当前pg的操作

17.es1
17.as0
17.8s1
17.4s2
17.3s1
17.3fs0
17.3ds0
17.3cs2
17.23s2
17.2es0      

PG相关操作

• 针对单个pg，列出当前pg的基本信息

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op info​

  ​输出如下

"pgid": "17.es1",
"last_update": "1380'5116",
"last_complete": "1380'5116",
"log_tail": "1376'4854",
"last_user_version": 5118,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [],
"history": {
    "epoch_created": 1290,
    "epoch_pool_created": 1290,
    "last_epoch_started": 1457,
    "last_interval_started": 1456,
    "last_epoch_clean": 1350,
    "last_interval_clean": 1349,
    "last_epoch_split": 0,
    "last_epoch_marked_full": 0,
    "same_up_since": 1456,
    "same_interval_since": 1456,
    "same_primary_since": 1290,
    "last_scrub": "0'0",
    "last_scrub_stamp": "2019-08-12 11:14:04.515869",
    "last_deep_scrub": "0'0",
    "last_deep_scrub_stamp": "2019-08-12 11:14:04.515869",
    "last_clean_scrub_stamp": "2019-08-12 11:14:04.515869"
    ...      

• 针对单个pg，列出当前pg元数据信息

  ​​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op log​

  ​​ 该命令的输出主要包括pg的历史版本信息，记录着每个对象的相关操作，以及当时​

  ​pg epoch​

  ​和对应的pg小版本

"pg_log_t": {
   "head": "1380'5116",
   "tail": "1376'4854",
   "log": [
       {
           "op": "modify",
           "object": "17:713bbf48:::rbd_data.18.cf83c74b0dc51.0000000000023ba0:head",
           "version": "1376'4855",
           "prior_version": "0'0",
           "reqid": "client.852676.0:164001",
           "extra_reqids": [],
           "mtime": "2019-08-12 14:40:08.571459",
           "return_code": 0,
           "mod_desc": {
               "object_mod_desc": {
                   "can_local_rollback": true,
                   "rollback_info_completed": true,
                   "ops": [
                       {
                           "code": "CREATE"
                       }
                   ]
               }
           }
       },
       ...      

打印pg中对象的元数据

​

​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op meta-list​

​

["meta",{"oid":"osdmap.1372","key":"","snapid":0,"hash":168878088,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.1053","key":"","snapid":0,"hash":168945672,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.1266","key":"","snapid":0,"hash":168892424,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.1101","key":"","snapid":0,"hash":168920072,"max":0,"pool":-1,"namespace":"","max":0}]
...      

• 将pg的元数据导出到指定文件，一般用作 备份

  ​​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op export --file /mnt/test.obj​

  ​

• 将指定的pg元数据导入到当前pg

  这里导入前需要先移除当前pg（remove之前请先export备份一下pg数据）

  ​​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op remove​

  ​​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op import --file /mnt/test.obj​

  ​
• 将集群中​

  ​incomplete​

  ​​的pg标记为​

  ​complete​

  ​​，这里需要根据集群的副本数将相关的所有osd上的pg进行标记

  ​​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op mark-complete​

  ​

对象相关操作

• 列出当前pg的所有对象

  ​​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op list​

  ​ 输出如下：

["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.000000000001bda5","key":"","snapid":-2,"hash":1725693966,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0}]
["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.0000000000033137","key":"","snapid":-2,"hash":1782448142,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0}]
["17.es1",{"oid":"rbd_data.18.cf83c74b0dc51.000000000003f128","key":"","snapid":-2,"hash":1246789646,"max":0,"pool":17,"namespace":"","shard_id":1,"max":0
...
...      

从以上获取的对象中取一个，打印对象的元数据信息

​

​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a dump​

​

{
"id": {
    "oid": "rbd_data.18.cf83c74b0dc51.000000000003925a",
    "key": "",
    "snapid": -2,
    "hash": 1962672078,
    "max": 0,
    "pool": 17,
    "namespace": "",
    "shard_id": 1,
    "max": 0
},
"info": {
    "oid": {
        "oid": "rbd_data.18.cf83c74b0dc51.000000000003925a",
        "key": "",
        "snapid": -2,
        "hash": 1962672078,
        "max": 0,
        "pool": 17,
        "namespace": ""
    },
    ...      

• 获取对象的属性信息

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore​

  ​

[root@node1 ceph]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a list-attrs
_
hinfo_key
snapset      

• 删除对象

  ​

  ​ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a remove​

  ​请慎重测试，如有pg备份，则可以进行测试

[root@node1 ceph]# ceph-objectstore-tool-bak --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore rbd_data.18.cf83c74b0dc51.000000000003925a remove
remove 1#17:73ffdf2e:::rbd_data.18.cf83c74b0dc51.000000000003925a:head#      

• 查看pg中是否还有该对象

  ​​

  ​ceph-objectstore-tool-bak --data-path /var/lib/ceph/osd/ceph-8/ --type bluestore --pgid 17.es1 --op list|grep rbd_data.18.cf83c74b0dc51.000000000003925a​

  ​并无输出

总结