#頭條創作挑戰賽#
目錄
- 1，使用docker編譯TensorFlow
- 2，隻要設定好cpu和記憶體即可，漫長等待編譯tensorflow
- 3，編譯完成，開始進行whl 安裝包
- 4，總結
1，使用docker編譯TensorFlow
直接參考goolge的中文官方文檔：
https://tensorflow.google.cn/install/docker?hl=zh-cn
這裡要特别的注意，帶 devel 是最新的 TensorFlow 源代碼建構僅支援 CPU 的軟體包。
裡面并沒有tensorflow的庫，特别的大。
docker pull tensorflow/tensorflow:devel
docker pull tensorflow/tensorflow:devel-gpu
docker run --name tfbuild -itd -v $PWD:/mnt \
-e HOST_PERMS="$(id -u):$(id -g)" tensorflow/tensorflow:devel
然後就是登陸進容器進行編譯了。這個按照官方網站的操作手冊一點一點執行就行了。
docker exec -it tfbuild bash
#選擇預設值就行。
#首先要切換到tags 的分支 2.11.0 ，預設是master是開發的版本，不是正式版本。
#切換分支代碼，master是開發版本
cd /tensorflow_src/
# 大工程分支特别多，耐心等待。
git fetch --tags
git checkout v2.11.0
# git status
HEAD detached at v2.11.0
# 一頓預設配置就行。
./configure
# 設定8G記憶體，太高會導緻killed
# https://github.com/tensorflow/tensorflow/issues/41480
# 這個是兩個斜杠！！好像是個參數的問題。目前目錄下有個檔案：
# /tensorflow_src/tensorflow/tools/pip_package/build_pip_package.sh
#
bazel build --config=opt --local_ram_resources=8000 --local_cpu_resources=4 //tensorflow/tools/pip_package:build_pip_package
# create package
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /mnt/out
chown $HOST_PERMS /tensorflow/out/tensorflow-version-tags.whl
這樣就完成了cpu的編譯優化，whl就是二進制檔案。然後直接放到容器中按照就行。
開始編譯中，漫長的等待。
因為在python的官網沒有 ubuntu的鏡像，是以直接用tensorflow的鏡像先解除安裝，再安裝就行了。
2，隻要設定好cpu和記憶體即可，漫長等待編譯tensorflow
time bazel build -c opt //tensorflow/tools/pip_package:build_pip_package \
--local_ram_resources=4000 --local_cpu_resources=4
# bazel build -c opt //tensorflow/tools/pip_package:build_pip_package --local_ram_resources=4000 --local_cpu_resources=4
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
Inherited 'common' options: --isatty=1 --terminal_columns=140
INFO: Reading rc options for 'build' from /tensorflow_src/.bazelrc:
Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /tensorflow_src/.bazelrc:
'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false
INFO: Reading rc options for 'build' from /tensorflow_src/.tf_configure.bazelrc:
'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3/dist-packages --python_path=/usr/bin/python3
INFO: Reading rc options for 'build' from /tensorflow_src/.bazelrc:
'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /tensorflow_src/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /tensorflow_src/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:linux in file /tensorflow_src/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-unknown-warning --copt=-Wno-array-parameter --copt=-Wno-stringop-overflow --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /tensorflow_src/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/4ce3e4da2e21ae4dfcee9366415e55f408c884ec.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/openxla/stablehlo/archive/fdd47908468488cbbb386bb7fc723dc19321cb83.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/e8f74a9763aa36559980a0c2f37f587794995622.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://golang.org/dl/?mode=json&include=all failed: class java.io.IOException connect timed out
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (544 packages loaded, 31179 targets configured).
INFO: Found 1 target...
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 12172.184s, Critical Path: 319.39s
INFO: 13069 processes: 1519 internal, 11550 local.
INFO: Build completed successfully, 13069 total actions
real 97m39.864s
user 0m0.620s
sys 0m0.355s
經過97分鐘的編譯終于可以了，然後再打包下；
# 打包生成 安裝檔案
# ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /mnt/out
check.warn(importable)
/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
Mon Mar 6 06:12:35 UTC 2023 : === Output wheel file is in: /mnt/out
3，編譯完成，開始進行whl 安裝包
測試tf 庫是否存在：
python -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
在沒有安裝前：
raise ImportError("Could not import tensorflow. Do not import tensorflow "
安裝之後：
mkdir /root/.pip/
# 增加 pip 的源，再進行安裝 tf
echo "[global]" > ~/.pip/pip.conf
echo "index-url = https://mirrors.aliyun.com/pypi/simple/" >> ~/.pip/pip.conf
echo "[install]" >> ~/.pip/pip.conf
echo "trusted-host=mirrors.aliyun.com" >> ~/.pip/pip.conf
# 直接安裝 tf的編譯後的whl 檔案和相關的依賴庫：
p3 install tensorflow-2.11.0-cp38-cp38-linux_x86_64.whl
python -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
2023-03-06 06:26:01.652690: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-06 06:26:03.657543: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
tf.Tensor(-155.91696, shape=(), dtype=float32)
4，總結
tesnsorflow使用 devel 進行支援源代碼的編譯，非常的友善。
裡面的babel 都配置好了，隻要設定好了記憶體，慢慢等待編譯結果就行了。
--local_ram_resources=4000
--local_cpu_resources=4
就可以直接在目前的鏡像中進行安裝即可。