一、现象
最近发现Mysql服务器磁盘IO一直很高
[[email protected] ~]# iostat -k -d -x 1 10
Linux 3.10.0-957.el7.x86_64 (push-5-221) 2019年07月05日 _x86_64_ (8 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.08 0.00 14.89 0.09 177.79 23.88 0.34 23.05 5.42 23.05 11.01 16.40
dm-0 0.00 0.00 0.00 0.18 0.09 3.62 39.49 0.07 380.21 5.59 387.46 12.69 0.24
dm-1 0.00 0.00 0.00 0.00 0.00 0.01 8.98 0.00 91.25 12.04 92.94 6.55 0.00
dm-2 0.00 0.00 0.00 14.79 0.00 174.15 23.56 0.33 22.27 8.84 22.27 10.96 16.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 83.00 0.00 650.50 15.67 2.55 30.66 0.00 30.66 12.02 99.80
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 83.00 0.00 636.50 15.34 2.55 30.66 0.00 30.66 12.02 99.80
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 88.00 0.00 684.50 15.56 2.60 29.42 0.00 29.42 11.35 99.90
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 87.00 0.00 677.00 15.56 2.60 29.76 0.00 29.76 11.49 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 87.00 0.00 687.50 15.80 2.48 28.77 0.00 28.77 11.51 100.10
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 88.00 0.00 699.00 15.89 2.48 28.48 0.00 28.48 11.36 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 88.00 0.00 756.50 17.19 2.48 28.22 0.00 28.22 11.35 99.90
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 86.00 0.00 740.00 17.21 2.48 28.87 0.00 28.87 11.62 99.90
%util已经接近100%了，IO性能严重瓶颈，这台服务器有两个Mysql实例，3306是主库，3307是另一台Mysql的从库，进入从库查看从库状态
mysql> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.28.5.223
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-223.000012
Read_Master_Log_Pos: 511363808
Relay_Log_File: push-5-221-relay-bin.000034
Relay_Log_Pos: 382425277
Relay_Master_Log_File: master-223.000012
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 382425062
Relay_Log_Space: 511364283
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 2345
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2233306
Master_UUID: 165a203c-8021-11e9-bc40-6c2b5992e592
Master_Info_File: /home/mysql-5.7.26/data/3307/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Reading event from the relay log
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
ERROR:
No query specified
mysql>
Seconds_Behind_Master: 2345 发现从库同步延时很高，通过iotop查看
Total DISK READ : 0.00 B/s | Total DISK WRITE : 683.85 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 721.19 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
7061 be/4 mysql 0.00 B/s 153.28 K/s 0.00 % 72.27 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307
1766 be/4 mysql 0.00 B/s 27.51 K/s 0.00 % 8.51 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2730 be/4 mysql 0.00 B/s 11.79 K/s 0.00 % 7.50 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2164 be/4 mysql 0.00 B/s 11.79 K/s 0.00 % 7.47 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
8932 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 7.41 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
1840 be/4 mysql 0.00 B/s 35.37 K/s 0.00 % 7.14 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
936 be/4 mysql 0.00 B/s 27.51 K/s 0.00 % 6.77 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
1296 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 6.31 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
1201 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 5.32 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2728 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 5.05 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2539 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 5.00 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2531 be/4 mysql 0.00 B/s 47.16 K/s 0.00 % 4.21 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
6895 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 3.63 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307
2303 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 3.54 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
461 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 3.40 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2523 be/4 mysql 0.00 B/s 3.93 K/s 0.00 % 3.36 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2520 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 3.35 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2316 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 2.85 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2537 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 2.81 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2731 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 2.42 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
8940 be/4 mysql 0.00 B/s 3.93 K/s 0.00 % 2.07 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2310 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 1.91 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
8935 be/4 mysql 0.00 B/s 157.21 K/s 0.00 % 1.41 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
2302 be/4 mysql 0.00 B/s 3.93 K/s 0.00 % 1.30 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
6899 be/4 mysql 0.00 B/s 31.44 K/s 0.00 % 0.93 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307
32593 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.61 % [kworker/3:1]
7060 be/4 mysql 0.00 B/s 27.51 K/s 0.00 % 0.00 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307
2535 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 0.00 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
13824 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd-udevd
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --system --deserialize 17
发现3307和3306实例占用IO最高，进入3307实例，查看sync_binlog变量
mysql> show variables like '%sync_binlog%'
-> ;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog | 1 |
+---------------+-------+
1 row in set (0.01 sec)
mysql>
发现设置的是1，也就是说每次事务提交都会将binlog的缓存写入磁盘，严重影响磁盘效率，将变量设置为1000,
mysql> set global sync_binlog=1000;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like '%sync_binlog%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog | 1000 |
+---------------+-------+
1 row in set (0.00 sec)
再查看innodb_flush_log_at_trx_commit参数
mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout | 1 |
| innodb_flush_log_at_trx_commit | 1 |
+--------------------------------+-------+
2 rows in set (0.00 sec)
设置为1，也就是说每次事务提交，都会将innodb日志缓存写入磁盘，对磁盘效率影响很大，将它设置为2，每次事务提交时mysql都会把log buffer的数据写入log file，但是flush(刷到磁盘)操作并不会同时进行。该模式下，MySQL会每秒执行一次 flush(刷到磁盘)操作
mysql> set global innodb_flush_log_at_trx_commit=2;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout | 1 |
| innodb_flush_log_at_trx_commit | 2 |
+--------------------------------+-------+
2 rows in set (0.00 sec)
此时再查看从库同步状态
mysql> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.28.5.223
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-223.000012
Read_Master_Log_Pos: 539225447
Relay_Log_File: push-5-221-relay-bin.000034
Relay_Log_Pos: 490154357
Relay_Master_Log_File: master-223.000012
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 490154142
Relay_Log_Space: 539225922
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 1378
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2233306
Master_UUID: 165a203c-8021-11e9-bc40-6c2b5992e592
Master_Info_File: /home/mysql-5.7.26/data/3307/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: System lock
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
ERROR:
No query specified
发现，同步延迟已经在很快缩小了，很快变为0，完全同步，这时通过iostat 查看io状态
[[email protected] ~]# iostat -k -d -x 1 10/s 0.00 % 0.00 % [kworker/2:0H]
Linux 3.10.0-957.el7.x86_64 (push-5-221) 2019年07月05日 _x86_64_ (8 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.08 0.00 14.91 0.09 178.29 23.92 0.34 23.05 5.42 23.06 11.01 16.43
dm-0 0.00 0.00 0.00 0.18 0.09 3.62 39.48 0.07 380.15 5.59 387.40 12.69 0.24
dm-1 0.00 0.00 0.00 0.00 0.00 0.01 8.97 0.00 91.84 12.04 93.52 6.63 0.00
dm-2 0.00 0.00 0.00 14.81 0.00 174.66 23.59 0.33 22.28 8.84 22.28 10.96 16.23
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 114.00 0.00 2013.00 35.32 3.14 27.40 0.00 27.40 8.75 99.80
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 113.00 0.00 2007.50 35.53 3.14 27.65 0.00 27.65 8.83 99.80
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 108.91 0.00 3026.24 55.57 3.10 27.85 0.00 27.85 9.05 98.51
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 111.88 0.00 3121.29 55.80 3.10 27.11 0.00 27.11 8.81 98.51
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 98.00 0.00 2648.00 54.04 2.81 29.63 0.00 29.63 9.71 95.20
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 95.00 0.00 2551.50 53.72 2.81 30.59 0.00 30.59 10.02 95.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 103.00 0.00 2037.00 39.55 2.97 28.67 0.00 28.67 9.12 93.90
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 104.00 0.00 2051.00 39.44 2.97 28.39 0.00 28.39 9.03 93.90
发现%util依旧差不多100%，磁盘IO性能还是很低，我们再把3306实例的变量设置改一下
[[email protected] ~]# mysql -uroot -p -S /var/lib/mysql/3306/mysql.sock
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1476252
Server version: 5.7.26-log MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show variables like '%sync_binlog%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog | 1 |
+---------------+-------+
1 row in set (0.01 sec)
mysql> set global sync_binlog=1000;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout | 1 |
| innodb_flush_log_at_trx_commit | 1 |
+--------------------------------+-------+
2 rows in set (0.00 sec)
mysql> set global innodb_flush_log_at_trx_commit=2;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout | 1 |
| innodb_flush_log_at_trx_commit | 2 |
+--------------------------------+-------+
2 rows in set (0.00 sec)
mysql>
此时，再通过iostat查看IO
[[email protected] ~]# iostat -k -d -x 1 10
Linux 3.10.0-957.el7.x86_64 (push-5-221) 2019年07月05日 _x86_64_ (8 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.08 0.00 14.92 0.09 178.36 23.92 0.34 23.06 5.42 23.06 11.01 16.43
dm-0 0.00 0.00 0.00 0.18 0.09 3.62 39.48 0.07 380.15 5.59 387.39 12.69 0.24
dm-1 0.00 0.00 0.00 0.00 0.00 0.01 8.97 0.00 91.84 12.04 93.52 6.63 0.00
dm-2 0.00 0.00 0.00 14.81 0.00 174.72 23.59 0.33 22.28 8.84 22.28 10.96 16.23
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 21.00 0.00 604.00 57.52 0.38 18.10 0.00 18.10 13.76 28.90
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 21.00 0.00 604.00 57.52 0.38 18.10 0.00 18.10 13.76 28.90
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 34.00 0.00 2220.50 130.62 1.29 37.88 0.00 37.88 12.88 43.80
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 34.00 0.00 2220.50 130.62 1.29 37.88 0.00 37.88 12.88 43.80
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 42.00 0.00 2056.00 97.90 1.29 30.67 0.00 30.67 9.40 39.50
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 42.00 0.00 2056.00 97.90 1.29 30.67 0.00 30.67 9.40 39.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 21.00 0.00 550.00 52.38 0.34 16.19 0.00 16.19 13.76 28.90
dm-0 0.00 0.00 0.00 2.00 0.00 1.50 1.50 0.05 24.00 0.00 24.00 24.00 4.80
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 19.00 0.00 548.50 57.74 0.29 15.37 0.00 15.37 12.74 24.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 18.00 0.00 524.00 58.22 0.34 18.61 0.00 18.61 13.33 24.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 18.00 0.00 524.00 58.22 0.34 18.61 0.00 18.61 13.33 24.00