環境
- Red Hat Enterprise Linux Server release 7
問題
Resource monitor time out occur when sub path of the multipath device fails
Error seen in "/var/log/messages"
Raw
Aug 16 13:19:21 xxxxx kernel: qla2xxx [0000:08:00.0]-801c:1: Abort command issued nexus=1:3:0 -- 1 2002.
Aug 16 13:19:44 xxxxx lrmd[4223]: warning: KEP_vg_arch_monitor_60000 process (PID 22244) timed out
決議
Tuning options with dm-multipath and in SCSI error handling which should reduce the time required in completing error handling on affected paths and then failover the IO to remaining available sub paths quickly. This should help to avoid monitor timeouts when only few of the sub paths to SAN devices are affected:
[1] Edit the "defaults" section in "/etc/multipath.conf" file to reduce the "checker_timeout", "polling_interval":
defaults {
checker_timeout 10
polling_interval 5
}
-
Then update the devices section to reduce "fast_io_fail_tmo", "dev_loss_tmo":
Raw
devices { device { vendor "3PARdata" product "VV" failback "immediate" rr_weight "uniform" no_path_retry 2 ### Edited rr_min_io_rq 1 fast_io_fail_tmo 5 ### Edited dev_loss_tmo 10 ### Edited } }
In the above snip, "no_path_retry" is reduced to 2, this is because in RHEL 7 the minimum value for "dev_loss_tmo" is calculated as a product of current "polling_interval" and "no_path_retry" for the devices. So to reduce the "dev_loss_tmo" for affected paths we would need to reduce the "no_path_retry" to 2.
After updating the above changes, please reload "multipathd" service to make these changes effective:
[2] Then set the following two tunings:
eh_timeout 5 Per device
eh_deadline 5 Overall cap on the error handler, only starts when the error handler kicks in.
To set above options:
$ cd /sys/block
for d in sd*
do
echo 5 > $d/device/eh_timeout
done
$ cd /sys/class/scsi_host
for host in host1 host2
do
echo 5 > $host/eh_deadline
done
根源
- Cluster node is having SAN devices connected through 2 Qlogic FC HBAs. And there were command timeouts, command aborts happening only for the sub paths connected through Qlogic FC HBA host1. So the sub paths through another HBA host2 were still available. As the sub paths through another HBA were still available we expect the monitor IO operation on above resources should not fail.
- SCSI error handling, and then IO failover from affected sub paths to the non-affected paths took slightly more time which was long enough to reach to the monitor timeout on resource. Due to this, a monitor operation on resources got timed out and pacemaker had initiated a recovery action on the resources.