TFServing Build and Install
Setup
CPU Base Docker Image
GCC Version |
Python Version |
IMAGE |
|---|---|---|
7.5.0 |
3.6.9 |
alideeprec/deeprec-base:deeprec-base-cpu-py36-ubuntu18.04 |
9.4.0 |
3.8.10 |
alideeprec/deeprec-base:deeprec-base-cpu-py38-ubuntu20.04 |
11.2.0 |
3.8.6 |
alideeprec/deeprec-base:deeprec-base-cpu-py38-ubuntu22.04 |
GPU Base Docker Image
GCC Version |
Python Version |
CUDA VERSION |
IMAGE |
|---|---|---|---|
7.5.0 |
3.6.9 |
CUDA 11.6.1 |
alideeprec/deeprec-base:deeprec-base-gpu-py36-cu116-ubuntu18.04 |
9.4.0 |
3.8.10 |
CUDA 11.6.2 |
alideeprec/deeprec-base:deeprec-base-gpu-py38-cu116-ubuntu20.04 |
11.2.0 |
3.8.6 |
CUDA 11.7.1 |
alideeprec/deeprec-base:deeprec-base-gpu-py38-cu117-ubuntu22.04 |
CPU Dev Docker (with bazel cache)
GCC Version |
Python Version |
IMAGE |
|---|---|---|
7.5.0 |
3.6.9 |
alideeprec/deeprec-build:deeprec-dev-cpu-py36-ubuntu18.04 |
9.4.0 |
3.8.10 |
alideeprec/deeprec-build:deeprec-dev-cpu-py38-ubuntu20.04 |
GPU(cuda11.6) Dev Docker (with bazel cache)
GCC Version |
Python Version |
CUDA VERSION |
IMAGE |
|---|---|---|---|
7.5.0 |
3.6.9 |
CUDA 11.6.1 |
alideeprec/deeprec-build:deeprec-dev-gpu-py36-cu116-ubuntu18.04 |
9.4.0 |
3.8.10 |
CUDA 11.6.2 |
alideeprec/deeprec-build:deeprec-dev-gpu-py38-cu116-ubuntu20.04 |
TFServing Source Code
We provide optimized TFServing which could highly improve performance in inference, such as SessionGroup, CUDA multi-stream, etc.
Source Code: https://github.com/DeepRec-AI/serving
Develop Branch: master, Latest Release Branch: deeprec2402
TFServing Build
Build Package Builder-CPU
bazel build -c opt tensorflow_serving/...
Build CPU Package Builder with OneDNN + Eigen Threadpool
bazel build -c opt --config=mkl_threadpool --define build_with_mkl_dnn_v1_only=true tensorflow_serving/...
Build Package Builder-GPU
bazel build -c opt --config=cuda tensorflow_serving/...
Build Package
bazel-bin/tensorflow_serving/tools/pip_package/build_pip_package /tmp/tf_serving_client_whl
Server Bin
Server Bin would generated in following directory:
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server