TensorFlow多GPU并行計算

1. TensorFlow指定特定GPU或者CPU進行計算：

說明：示例計算機為單CPU（編号為0），單GPU（編号為0），安裝的TensorFlow為GPU版。

本文的結構如下：

• 預設為GPU #0
• 指定CPU #0
• 指定GPU #1
• 指定GPU #0 + CPU #0

1.1 預設為GPU #0

I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcurand.so locally
           

In []: with tf.Session() as sess:
   ...:     matrix1=tf.constant([[,]])
   ...:     matrix2=tf.constant([[],[]])
   ...:     product=tf.matmul(matrix1,matrix2)
   ...:     result=sess.run(product)
   ...:     print result
   ...:
           

I tensorflow/core/common_runtime/gpu/gpu_init.cc:] Found device  with properties: 
name: GeForce GTX 
major:  minor:  memoryClockRate (GHz) 
pciBusID ::
Total memory: GiB
Free memory: GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] DMA:  
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] :   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:] Creating TensorFlow device (/gpu:) -> (device: , name: GeForce GTX , pci bus id: ::)
[[ 12.]]
           

1.2 指定GPU #0

In []: import tensorflow as tf
           

I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcurand.so locally
           

In []: with tf.Session() as sess:
   ...:     with tf.device("/gpu:0"):
   ...:         matrix1=tf.constant([[,]])
   ...:         matrix2=tf.constant([[],[]])
   ...:         product=tf.matmul(matrix1,matrix2)
   ...:     result=sess.run(product)
   ...:     print result
   ...:
           

I tensorflow/core/common_runtime/gpu/gpu_init.cc:] Found device  with properties: 
name: GeForce GTX 
major:  minor:  memoryClockRate (GHz) 
pciBusID ::
Total memory: GiB
Free memory: GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] DMA:  
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] :   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:] Creating TensorFlow device (/gpu:) -> (device: , name: GeForce GTX , pci bus id: ::)
[[ 12.]]
           

1.3 指定CPU #0

I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcurand.so locally
           

In []: with tf.Session() as sess:
   ...:     with tf.device("/cpu:0"):
   ...:         matrix1=tf.constant([[,]])
   ...:         matrix2=tf.constant([[],[]])
   ...:         product=tf.matmul(matrix1,matrix2)
   ...:     result=sess.run(product)
   ...:     print result
   ...:
           

I tensorflow/core/common_runtime/gpu/gpu_init.cc:] Found device  with properties: 
name: GeForce GTX 
major:  minor:  memoryClockRate (GHz) 
pciBusID ::
Total memory: GiB
Free memory: GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] DMA:  
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] :   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:] Creating TensorFlow device (/gpu:) -> (device: , name: GeForce GTX , pci bus id: ::)
[[ 12.]]
           

1.4 指定GPU #1

In []: with tf.Session() as sess:
   ...:     with tf.device("/gpu:1"):
   ...:         matrix1=tf.constant([[,]])
   ...:         matrix2=tf.constant([[],[]])
   ...:         product=tf.matmul(matrix1,matrix2)
   ...:     result=sess.run(product)
   ...:     print result
   ...:
           

I tensorflow/core/common_runtime/gpu/gpu_device.cc:] Creating TensorFlow device (/gpu:) -> (device: , name: GeForce GTX , pci bus id: ::)
           

InvalidArgumentError     Traceback (most recent call last)

<ipython-input--ab0827> in <module>()
               matrix2=tf.constant([[2.],[2.]])
               product=tf.matmul(matrix1,matrix2)
----> 6     result=sess.run(product)
           print result
      
           

InvalidArgumentError: **Cannot assign a device to node** 'MatMul_2': 
Could not satisfy explicit device specification '/device:GPU:1' 
because no devices matching that specification are registered in this process; 
available devices: /job:localhost/replica:/task:/cpu:, /job:localhost/replica:/task:/gpu:
[[Node: MatMul_2 = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/device:GPU:1"](Const_4, Const_5)]]
           

說明：因為本機隻有一塊GPU，編号為0,而我指定該計算在GPU:1中進行，才報錯。

1.4 指定GPU #0 + GPU #0

I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcurand.so locally
           

In []: with tf.Session() as sess:
   ...:     with tf.device("/cpu:0"):
   ...:         matrix1=tf.constant([[,]])
   ...:         matrix2=tf.constant([[],[]])
   ...:     with tf.device("/gpu:0"):
   ...:         product=tf.matmul(matrix1,matrix2)
   ...:     result=sess.run(product)
   ...:     print result
   ...:
           

I tensorflow/core/common_runtime/gpu/gpu_init.cc:] Found device  with properties: 
name: GeForce GTX 
major:  minor:  memoryClockRate (GHz) 
pciBusID ::
Total memory: GiB
Free memory: GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] DMA:  
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] :   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:] Creating TensorFlow device (/gpu:) -> (device: , name: GeForce GTX , pci bus id: ::)
[[ 12.]]
           

當然，也可以直接這麼寫：

I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:] successfully opened CUDA library libcurand.so locally
           

In []: with tf.device("/cpu:0"):
   ...:     matrix1=tf.constant([[,]])
   ...:     matrix2=tf.constant([[],[]])
   ...:

In []: with tf.device("/gpu:0"):
   ...:     product=tf.matmul(matrix1,matrix2)
   ...:

In []: sess=tf.Session()
           

I tensorflow/core/common_runtime/gpu/gpu_init.cc:] Found device  with properties: 
name: GeForce GTX 
major:  minor:  memoryClockRate (GHz) 
pciBusID ::
Total memory: GiB
Free memory: GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] DMA:  
I tensorflow/core/common_runtime/gpu/gpu_init.cc:] :   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:] Creating TensorFlow device (/gpu:) -> (device: , name: GeForce GTX , pci bus id: ::)
           

In []: result=sess.run(product)

In []: print result
[[ 12.]]
           

In []: sess.close()
           

注意：将節點operation放到 with tf.device(..): 裡面,而啟動語句或者不需要計算資源的語句放到with的外面