如何监控GPU状态------gpustat

最近用上了GPU，但是呢，不是自己在用，得需要监视下使用状态吧，所以就有了gpustat。

1、安装gpustat

pip install gpustat
           

2、使用

gpustat -cpu
           

Options:

--color :

Force colored output (even when stdout is not a tty)

--no-color :

Suppress colored output

-u, --show-user :

Display username of the process owner

-c, --show-cmd :

Display the process name

-p, --show-pid :

Display PID of the process

-P, --show-power :

Display GPU power usage and/or limit (draw or draw,limit)

--watch, -i, --interval :

Run in watch mode (equivalent to watch gpustat) if given. Denotes interval between updates. (#41)

--json :

JSON Output (Experimental, #10)

Tips

To periodically watch, try

gpustat --watch

or

gpustat -i

(#41).

For older versions, one may use

watch --color -n1.0 gpustat --color

。

Running

nvidia-smi daemon

(root privilege required) will make the query much faster and use less CPU (#54).

The GPU ID (index) shown by

gpustat

(and

nvidia-smi

) is PCI BUS ID, while CUDA differently assigns the fastest GPU with the lowest ID by default. Therefore, in order to make CUDA and

gpustat

use same GPU index, configure the

CUDA_DEVICE_ORDER

environment variable to

PCI_BUS_ID

(before setting

CUDA_VISIBLE_DEVICES

for your CUDA program):

export CUDA_DEVICE_ORDER=PCI_BUS_ID

.

效果图：