模型部署框架类型
算法模型的部署主要可以分成两个方面。一是在移动端/边缘端的部署，即嵌入式，通常以SDK形式呈现。另一个是云端/服务端，通常以服务的形式呈现；今天着重聊聊部署流程，后续移动端部署、具体厂商的智能硬件部署、云端server会开专题介绍。
边缘端
模型训练:通过pytorch、tensorflow等深度学习框架进行训练算法模型，得到模型权重文件，模型训练部分今天不着重介绍，后续专题会展开讨论训练tricks、模型调优、模型剪枝、蒸馏、量化。
模型转化:把权重文件转为对应智能硬件的形态，方便利用对应的GPU、NPU或者IPU智能硬件加速单元来达到加速效果。
算法部署:依照原模型算法推理逻辑对应实现在嵌入式端。
模型转化
包括英伟达、⾼通、华为、AMD在内的⼚家，都在神经⽹络加速⽅⾯投⼊了研发⼒量。通过量化、裁剪和压缩来降低模型尺⼨。更快的推断可以通过在降低精度的前提下使⽤⾼效计算平台⽽达到，其中包括intel MKL-DNN，ARM CMSIS，Qualcomm SNPE，Nvidia TensorRT，海思、RockChip RKNN，SigmarStar SGS_IPU等。
依TensorRT为例,其他平台的部署系列后面会出详细手把手教程。
TensorRT
方式一：把训练得到的权重文件如(pt,pb)先转化为Onnx形式，使用onnx-simplifier对模型进行图优化，得到一个简洁明了的模型图，最后通过trtexec转为对应的engine文件。
以Yolov5为例，导出onnx代码
# YOLOv5 ONNX export
try:
check_requirements(('onnx',))
import onnx
LOGGER.info(f'\n{prefix} starting export with onnx {onnx.__version__}...')
f = file.with_suffix('.onnx')
torch.onnx.export(model, im, f, verbose=False, opset_version=opset,
training=torch.onnx.TrainingMode.TRAINING if train else torch.onnx.TrainingMode.EVAL,
do_constant_folding=not train,
input_names=['images'],
output_names=['output'],
dynamic_axes={'images': {0: 'batch', 2: 'height', 3: 'width'}, # shape(1,3,640,640)
'output': {0: 'batch', 1: 'anchors'} # shape(1,25200,85)
} if dynamic else None)
# Checks
model_onnx = onnx.load(f) # load onnx model
onnx.checker.check_model(model_onnx) # check onnx model
# LOGGER.info(onnx.helper.printable_graph(model_onnx.graph)) # print
# Simplify
if simplify:
try:
check_requirements(('onnx-simplifier',))
import onnxsim
LOGGER.info(f'{prefix} simplifying with onnx-simplifier {onnxsim.__version__}...')
model_onnx, check = onnxsim.simplify(
model_onnx,
dynamic_input_shape=dynamic,
input_shapes={'images': list(im.shape)} if dynamic else None)
assert check, 'assert check failed'
onnx.save(model_onnx, f)
except Exception as e:
LOGGER.info(f'{prefix} simplifier failure: {e}')
LOGGER.info(f'{prefix} export success, saved as {f} ({file_size(f):.1f} MB)')
return f
except Exception as e:
LOGGER.info(f'{prefix} export failure: {e}') 导出后的Onnx模型图
Yolov5-Onnx-结构图
然后执行
trtexec --onnx=weights/yolov5s.onnx --saveEngine=weights/yolov5s.engine 对于YOLOV5，官方已经提供了一键转各种格式的脚本，具体参考
在此仅提供模型转化的方法思路。
方式二：根据TensorRT官方API文档，手动搭建模型结构，最后根据API接口把模型转成engine文件。
同样的依照Yolov5为例：
提取模型权重
import sys
import argparse
import os
import struct
import torch
from utils.torch_utils import select_device
def parse_args():
parser = argparse.ArgumentParser(description='Convert .pt file to .wts')
parser.add_argument('-w', '--weights', required=True,
help='Input weights (.pt) file path (required)')
parser.add_argument(
'-o', '--output', help='Output (.wts) file path (optional)')
parser.add_argument(
'-t', '--type', type=str, default='detect', choices=['detect', 'cls', 'seg'],
help='determines the model is detection/classification')
args = parser.parse_args()
if not os.path.isfile(args.weights):
raise SystemExit('Invalid input file')
if not args.output:
args.output = os.path.splitext(args.weights)[0] + '.wts'
elif os.path.isdir(args.output):
args.output = os.path.join(
args.output,
os.path.splitext(os.path.basename(args.weights))[0] + '.wts')
return args.weights, args.output, args.type
pt_file, wts_file, m_type = parse_args()
print(f'Generating .wts for {m_type} model')
# Load model
print(f'Loading {pt_file}')
device = select_device('cpu')
model = torch.load(pt_file, map_location=device) # Load FP32 weights
model = model['ema' if model.get('ema') else 'model'].float()
if m_type in ['detect', 'seg']:
# update anchor_grid info
anchor_grid = model.model[-1].anchors * model.model[-1].stride[..., None, None]
# model.model[-1].anchor_grid = anchor_grid
delattr(model.model[-1], 'anchor_grid') # model.model[-1] is detect layer
# The parameters are saved in the OrderDict through the "register_buffer" method, and then saved to the weight.
model.model[-1].register_buffer("anchor_grid", anchor_grid)
model.model[-1].register_buffer("strides", model.model[-1].stride)
model.to(device).eval()
print(f'Writing into {wts_file}')
with open(wts_file, 'w') as f:
f.write('{}\n'.format(len(model.state_dict().keys())))
for k, v in model.state_dict().items():
vr = v.reshape(-1).cpu().numpy()
f.write('{} {} '.format(k, len(vr)))
for vv in vr:
f.write(' ')
f.write(struct.pack('>f', float(vv)).hex())
f.write('\n') 根据API接口构建编译Yolov5模型结构
核心代码块
ICudaEngine* build_engine(unsigned int maxBatchSize, IBuilder* builder, IBuilderConfig* config, DataType dt, float& gd, float& gw, std::string& wts_name) {
INetworkDefinition* network = builder->createNetworkV2(0U);
// Create input tensor of shape {3, INPUT_H, INPUT_W} with name INPUT_BLOB_NAME
ITensor* data = network->addInput(INPUT_BLOB_NAME, dt, Dims3{ 3, INPUT_H, INPUT_W });
assert(data);
std::map<std::string, Weights> weightMap = loadWeights(wts_name);
/* ------ yolov5 backbone------ */
auto conv0 = convBlock(network, weightMap, *data, get_width(64, gw), 6, 2, 1, "model.0");
assert(conv0);
auto conv1 = convBlock(network, weightMap, *conv0->getOutput(0), get_width(128, gw), 3, 2, 1, "model.1");
auto bottleneck_CSP2 = C3(network, weightMap, *conv1->getOutput(0), get_width(128, gw), get_width(128, gw), get_depth(3, gd), true, 1, 0.5, "model.2");
auto conv3 = convBlock(network, weightMap, *bottleneck_CSP2->getOutput(0), get_width(256, gw), 3, 2, 1, "model.3");
auto bottleneck_csp4 = C3(network, weightMap, *conv3->getOutput(0), get_width(256, gw), get_width(256, gw), get_depth(6, gd), true, 1, 0.5, "model.4");
auto conv5 = convBlock(network, weightMap, *bottleneck_csp4->getOutput(0), get_width(512, gw), 3, 2, 1, "model.5");
auto bottleneck_csp6 = C3(network, weightMap, *conv5->getOutput(0), get_width(512, gw), get_width(512, gw), get_depth(9, gd), true, 1, 0.5, "model.6");
auto conv7 = convBlock(network, weightMap, *bottleneck_csp6->getOutput(0), get_width(1024, gw), 3, 2, 1, "model.7");
auto bottleneck_csp8 = C3(network, weightMap, *conv7->getOutput(0), get_width(1024, gw), get_width(1024, gw), get_depth(3, gd), true, 1, 0.5, "model.8");
auto spp9 = SPPF(network, weightMap, *bottleneck_csp8->getOutput(0), get_width(1024, gw), get_width(1024, gw), 5, "model.9");
/* ------ yolov5 head ------ */
auto conv10 = convBlock(network, weightMap, *spp9->getOutput(0), get_width(512, gw), 1, 1, 1, "model.10");
auto upsample11 = network->addResize(*conv10->getOutput(0));
assert(upsample11);
upsample11->setResizeMode(ResizeMode::kNEAREST);
upsample11->setOutputDimensions(bottleneck_csp6->getOutput(0)->getDimensions());
ITensor* inputTensors12[] = { upsample11->getOutput(0), bottleneck_csp6->getOutput(0) };
auto cat12 = network->addConcatenation(inputTensors12, 2);
auto bottleneck_csp13 = C3(network, weightMap, *cat12->getOutput(0), get_width(1024, gw), get_width(512, gw), get_depth(3, gd), false, 1, 0.5, "model.13");
auto conv14 = convBlock(network, weightMap, *bottleneck_csp13->getOutput(0), get_width(256, gw), 1, 1, 1, "model.14");
auto upsample15 = network->addResize(*conv14->getOutput(0));
assert(upsample15);
upsample15->setResizeMode(ResizeMode::kNEAREST);
upsample15->setOutputDimensions(bottleneck_csp4->getOutput(0)->getDimensions());
ITensor* inputTensors16[] = { upsample15->getOutput(0), bottleneck_csp4->getOutput(0) };
auto cat16 = network->addConcatenation(inputTensors16, 2);
auto bottleneck_csp17 = C3(network, weightMap, *cat16->getOutput(0), get_width(512, gw), get_width(256, gw), get_depth(3, gd), false, 1, 0.5, "model.17");
/* ------ detect ------ */
IConvolutionLayer* det0 = network->addConvolutionNd(*bottleneck_csp17->getOutput(0), 3 * (Yolo::CLASS_NUM + 5), DimsHW{ 1, 1 }, weightMap["model.24.m.0.weight"], weightMap["model.24.m.0.bias"]);
auto conv18 = convBlock(network, weightMap, *bottleneck_csp17->getOutput(0), get_width(256, gw), 3, 2, 1, "model.18");
ITensor* inputTensors19[] = { conv18->getOutput(0), conv14->getOutput(0) };
auto cat19 = network->addConcatenation(inputTensors19, 2);
auto bottleneck_csp20 = C3(network, weightMap, *cat19->getOutput(0), get_width(512, gw), get_width(512, gw), get_depth(3, gd), false, 1, 0.5, "model.20");
IConvolutionLayer* det1 = network->addConvolutionNd(*bottleneck_csp20->getOutput(0), 3 * (Yolo::CLASS_NUM + 5), DimsHW{ 1, 1 }, weightMap["model.24.m.1.weight"], weightMap["model.24.m.1.bias"]);
auto conv21 = convBlock(network, weightMap, *bottleneck_csp20->getOutput(0), get_width(512, gw), 3, 2, 1, "model.21");
ITensor* inputTensors22[] = { conv21->getOutput(0), conv10->getOutput(0) };
auto cat22 = network->addConcatenation(inputTensors22, 2);
auto bottleneck_csp23 = C3(network, weightMap, *cat22->getOutput(0), get_width(1024, gw), get_width(1024, gw), get_depth(3, gd), false, 1, 0.5, "model.23");
IConvolutionLayer* det2 = network->addConvolutionNd(*bottleneck_csp23->getOutput(0), 3 * (Yolo::CLASS_NUM + 5), DimsHW{ 1, 1 }, weightMap["model.24.m.2.weight"], weightMap["model.24.m.2.bias"]);
auto yolo = addYoLoLayer(network, weightMap, "model.24", std::vector<IConvolutionLayer*>{det0, det1, det2});
yolo->getOutput(0)->setName(OUTPUT_BLOB_NAME);
network->markOutput(*yolo->getOutput(0));
// Build engine
builder->setMaxBatchSize(maxBatchSize);
config->setMaxWorkspaceSize(16 * (1 << 20)); // 16MB
#if defined(USE_FP16)
config->setFlag(BuilderFlag::kFP16);
#elif defined(USE_INT8)
std::cout << "Your platform support int8: " << (builder->platformHasFastInt8() ? "true" : "false") << std::endl;
assert(builder->platformHasFastInt8());
config->setFlag(BuilderFlag::kINT8);
Int8EntropyCalibrator2* calibrator = new Int8EntropyCalibrator2(1, INPUT_W, INPUT_H, "./coco_calib/", "int8calib.table", INPUT_BLOB_NAME);
config->setInt8Calibrator(calibrator);
#endif
std::cout << "Building engine, please wait for a while..." << std::endl;
ICudaEngine* engine = builder->buildEngineWithConfig(*network, *config);
std::cout << "Build engine successfully!" << std::endl;
// Don't need the network any more
network->destroy();
// Release host memory
for (auto& mem : weightMap) {
free((void*)(mem.second.values));
}
return engine;
} 具体参照
方式三：跟方式一一样先转成onnx图模型，根据TensorRT-onnx_parser模型转成engine文件。
核心代码块
bool compile(
Mode mode,
unsigned int maxBatchSize,
const ModelSource& source,
const CompileOutput& saveto,
std::vector<InputDims> inputsDimsSetup,
Int8Process int8process,
const std::string& int8ImageDirectory,
const std::string& int8EntropyCalibratorFile,
const size_t maxWorkspaceSize) {
if (mode == Mode::INT8 && int8process == nullptr) {
INFOE("int8process must not nullptr, when in int8 mode.");
return false;
}
bool hasEntropyCalibrator = false;
vector<uint8_t> entropyCalibratorData;
vector<string> entropyCalibratorFiles;
if (mode == Mode::INT8) {
if (!int8EntropyCalibratorFile.empty()) {
if (iLogger::exists(int8EntropyCalibratorFile)) {
entropyCalibratorData = iLogger::load_file(int8EntropyCalibratorFile);
if (entropyCalibratorData.empty()) {
INFOE("entropyCalibratorFile is set as: %s, but we read is empty.", int8EntropyCalibratorFile.c_str());
return false;
}
hasEntropyCalibrator = true;
}
}
if (hasEntropyCalibrator) {
if (!int8ImageDirectory.empty()) {
INFOW("imageDirectory is ignore, when entropyCalibratorFile is set");
}
}
else {
if (int8process == nullptr) {
INFOE("int8process must be set. when Mode is '%s'", mode_string(mode));
return false;
}
entropyCalibratorFiles = iLogger::find_files(int8ImageDirectory, "*.jpg;*.png;*.bmp;*.jpeg;*.tiff");
if (entropyCalibratorFiles.empty()) {
INFOE("Can not find any images(jpg/png/bmp/jpeg/tiff) from directory: %s", int8ImageDirectory.c_str());
return false;
}
if(entropyCalibratorFiles.size() < maxBatchSize){
INFOW("Too few images provided, %d[provided] < %d[max batch size], image copy will be performed", entropyCalibratorFiles.size(), maxBatchSize);
int old_size = entropyCalibratorFiles.size();
for(int i = old_size; i < maxBatchSize; ++i)
entropyCalibratorFiles.push_back(entropyCalibratorFiles[i % old_size]);
}
}
}
else {
if (hasEntropyCalibrator) {
INFOW("int8EntropyCalibratorFile is ignore, when Mode is '%s'", mode_string(mode));
}
}
INFO("Compile %s %s.", mode_string(mode), source.descript().c_str());
shared_ptr<IBuilder> builder(createInferBuilder(gLogger), destroy_nvidia_pointer<IBuilder>);
if (builder == nullptr) {
INFOE("Can not create builder.");
return false;
}
shared_ptr<IBuilderConfig> config(builder->createBuilderConfig(), destroy_nvidia_pointer<IBuilderConfig>);
if (mode == Mode::FP16) {
if (!builder->platformHasFastFp16()) {
INFOW("Platform not have fast fp16 support");
}
config->setFlag(BuilderFlag::kFP16);
}
else if (mode == Mode::INT8) {
if (!builder->platformHasFastInt8()) {
INFOW("Platform not have fast int8 support");
}
config->setFlag(BuilderFlag::kINT8);
}
shared_ptr<INetworkDefinition> network;
//shared_ptr<ICaffeParser> caffeParser;
shared_ptr<nvonnxparser::IParser> onnxParser;
if(source.type() == ModelSourceType::OnnX || source.type() == ModelSourceType::OnnXData){
const auto explicitBatch = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
network = shared_ptr<INetworkDefinition>(builder->createNetworkV2(explicitBatch), destroy_nvidia_pointer<INetworkDefinition>);
vector<nvinfer1::Dims> dims_setup(inputsDimsSetup.size());
for(int i = 0; i < inputsDimsSetup.size(); ++i){
auto s = inputsDimsSetup[i];
dims_setup[i] = convert_to_trt_dims(s.dims());
dims_setup[i].d[0] = -1;
}
//from onnx is not markOutput
onnxParser.reset(nvonnxparser::createParser(*network, gLogger, dims_setup), destroy_nvidia_pointer<nvonnxparser::IParser>);
if (onnxParser == nullptr) {
INFOE("Can not create parser.");
return false;
}
if(source.type() == ModelSourceType::OnnX){
if (!onnxParser->parseFromFile(source.onnxmodel().c_str(), 1)) {
INFOE("Can not parse OnnX file: %s", source.onnxmodel().c_str());
return false;
}
}else{
if (!onnxParser->parseFromData(source.onnx_data(), source.onnx_data_size(), 1)) {
INFOE("Can not parse OnnX file: %s", source.onnxmodel().c_str());
return false;
}
}
}
else {
INFOE("not implementation source type: %d", source.type());
Assert(false);
}
set_layer_hook_reshape(nullptr);
auto inputTensor = network->getInput(0);
auto inputDims = inputTensor->getDimensions();
shared_ptr<Int8EntropyCalibrator> int8Calibrator;
if (mode == Mode::INT8) {
auto calibratorDims = inputDims;
calibratorDims.d[0] = maxBatchSize;
if (hasEntropyCalibrator) {
INFO("Using exist entropy calibrator data[%d bytes]: %s", entropyCalibratorData.size(), int8EntropyCalibratorFile.c_str());
int8Calibrator.reset(new Int8EntropyCalibrator(
entropyCalibratorData, calibratorDims, int8process
));
}
else {
INFO("Using image list[%d files]: %s", entropyCalibratorFiles.size(), int8ImageDirectory.c_str());
int8Calibrator.reset(new Int8EntropyCalibrator(
entropyCalibratorFiles, calibratorDims, int8process
));
}
config->setInt8Calibrator(int8Calibrator.get());
}
INFO("Input shape is %s", join_dims(vector<int>(inputDims.d, inputDims.d + inputDims.nbDims)).c_str());
INFO("Set max batch size = %d", maxBatchSize);
INFO("Set max workspace size = %.2f MB", maxWorkspaceSize / 1024.0f / 1024.0f);
INFO("Base device: %s", CUDATools::device_description().c_str());
int net_num_input = network->getNbInputs();
INFO("Network has %d inputs:", net_num_input);
vector<string> input_names(net_num_input);
for(int i = 0; i < net_num_input; ++i){
auto tensor = network->getInput(i);
auto dims = tensor->getDimensions();
auto dims_str = join_dims(vector<int>(dims.d, dims.d+dims.nbDims));
INFO(" %d.[%s] shape is %s", i, tensor->getName(), dims_str.c_str());
input_names[i] = tensor->getName();
}
int net_num_output = network->getNbOutputs();
INFO("Network has %d outputs:", net_num_output);
for(int i = 0; i < net_num_output; ++i){
auto tensor = network->getOutput(i);
auto dims = tensor->getDimensions();
auto dims_str = join_dims(vector<int>(dims.d, dims.d+dims.nbDims));
INFO(" %d.[%s] shape is %s", i, tensor->getName(), dims_str.c_str());
}
int net_num_layers = network->getNbLayers();
INFO("Network has %d layers:", net_num_layers);
for(int i = 0; i < net_num_layers; ++i){
auto layer = network->getLayer(i);
auto name = layer->getName();
auto type_str = layer_type_name(layer);
auto input0 = layer->getInput(0);
if(input0 == nullptr) continue;
auto output0 = layer->getOutput(0);
auto input_dims = input0->getDimensions();
auto output_dims = output0->getDimensions();
bool has_input = layer_has_input_tensor(layer);
bool has_output = layer_has_output_tensor(layer);
auto descript = layer_descript(layer);
type_str = iLogger::align_blank(type_str, 18);
auto input_dims_str = iLogger::align_blank(dims_str(input_dims), 18);
auto output_dims_str = iLogger::align_blank(dims_str(output_dims), 18);
auto number_str = iLogger::align_blank(format("%d.", i), 4);
const char* token = " ";
if(has_input)
token = " >>> ";
else if(has_output)
token = " *** ";
INFOV("%s%s%s %s-> %s%s", token,
number_str.c_str(),
type_str.c_str(),
input_dims_str.c_str(),
output_dims_str.c_str(),
descript.c_str()
);
}
builder->setMaxBatchSize(maxBatchSize);
config->setMaxWorkspaceSize(maxWorkspaceSize);
auto profile = builder->createOptimizationProfile();
for(int i = 0; i < net_num_input; ++i){
auto input = network->getInput(i);
auto input_dims = input->getDimensions();
input_dims.d[0] = 1;
profile->setDimensions(input->getName(), nvinfer1::OptProfileSelector::kMIN, input_dims);
profile->setDimensions(input->getName(), nvinfer1::OptProfileSelector::kOPT, input_dims);
input_dims.d[0] = maxBatchSize;
profile->setDimensions(input->getName(), nvinfer1::OptProfileSelector::kMAX, input_dims);
}
// not need
// for(int i = 0; i < net_num_output; ++i){
// auto output = network->getOutput(i);
// auto output_dims = output->getDimensions();
// output_dims.d[0] = 1;
// profile->setDimensions(output->getName(), nvinfer1::OptProfileSelector::kMIN, output_dims);
// profile->setDimensions(output->getName(), nvinfer1::OptProfileSelector::kOPT, output_dims);
// output_dims.d[0] = maxBatchSize;
// profile->setDimensions(output->getName(), nvinfer1::OptProfileSelector::kMAX, output_dims);
// }
config->addOptimizationProfile(profile);
// error on jetson
// auto timing_cache = shared_ptr<nvinfer1::ITimingCache>(config->createTimingCache(nullptr, 0), [](nvinfer1::ITimingCache* ptr){ptr->reset();});
// config->setTimingCache(*timing_cache, false);
// config->setFlag(BuilderFlag::kGPU_FALLBACK);
// config->setDefaultDeviceType(DeviceType::kDLA);
// config->setDLACore(0);
INFO("Building engine...");
auto time_start = iLogger::timestamp_now();
shared_ptr<ICudaEngine> engine(builder->buildEngineWithConfig(*network, *config), destroy_nvidia_pointer<ICudaEngine>);
if (engine == nullptr) {
INFOE("engine is nullptr");
return false;
}
if (mode == Mode::INT8) {
if (!hasEntropyCalibrator) {
if (!int8EntropyCalibratorFile.empty()) {
INFO("Save calibrator to: %s", int8EntropyCalibratorFile.c_str());
iLogger::save_file(int8EntropyCalibratorFile, int8Calibrator->getEntropyCalibratorData());
}
else {
INFO("No set entropyCalibratorFile, and entropyCalibrator will not save.");
}
}
}
INFO("Build done %lld ms !", iLogger::timestamp_now() - time_start);
// serialize the engine, then close everything down
shared_ptr<IHostMemory> seridata(engine->serialize(), destroy_nvidia_pointer<IHostMemory>);
if(saveto.type() == CompileOutputType::File){
return iLogger::save_file(saveto.file(), seridata->data(), seridata->size());
}else{
((CompileOutput&)saveto).set_data(vector<uint8_t>((uint8_t*)seridata->data(), (uint8_t*)seridata->data()+seridata->size()));
return true;
}
}
}; //namespace TRTBuilder 具体参照
至此模型转化这部分完成。
三种方式的优缺点：
方式一、方式三相对于方式二更为简单方便快捷，特别方式一零代码即可实现模型的转化，反观方式二需要清晰模型结构，清晰API接口算子并手撸代码完成构建engine。但方式一、方式三对于一些模型如transform、Vit模型由于一些算子还未支持，故不能一键转化，而方式二则可完成，总体来说方式二相比其他更为灵活，但上手难度更大。
算法部署
整体流程
流程图
输入:着重说下视频流如rtsp、webrtc、rtmp这种实时视频流，我们需要先对流进行解码从而得到RGB图像(YUV420、NV12、NV21 -> RGB)，其中解码又分为软解码和硬解码，软解码如libx264,libx265等，硬解码如Nvidia的CUVID以及海思，RockChip的Mpp等，关于视频流的编解码后续会开专题详细介绍。
预处理:把得到的RGB图像依照跟训练时进行同样的预处理，如Yolov5需要自适应缩放、归一化操作；人脸检测scrfd需要自适应缩放、减均值127.5，除方差128等操作；对于自适应缩放可以采用仿射变换、letterbox的形式实现；对于减均值、除方差，NVIDIA可以采用CUDA进行操作，从而达到提速的效果。
模型推理：把经过上边两步的图像data送进序列化好的engine进行model_forward，得到output_tensor。
后处理：把上述得到的output_tensor，进行后处理decode,依照目标检测为例这个操作一般为general_anchor、nms、iou，坐标映射到原图(transform_pred)等操作；分类模型则一般为get_max_pred；姿态识别模型一般为keypoints_from_heatmap、transform_pred等。
输出:经过后处理后，就得到了最终的输出结果，如检测项，分类类别，keypoints，人脸坐标等等，最终可根据实际场景进行告警推送等应用开发，或者把告警图片进行编码(RGB->YUV420)以视频流的方式推送到流媒体服务器。
生成SDK
对于Hisi3516、3519或者rv1126、rv1109这类平台，flash空间小，需要交叉编译，可打包成动态链接库，提供接口函数供上层应用调用；对于rv3399、rk3568、jetson产品自带Ubuntu或者Linaro系统，可终端机自行编译，并且可部署python，可利用pybind11进行衔接交互。
云服务端
关于模型的云端部署，业界也有许多开源的解决方案，但目前为止来看，还没有一种真的可以一统业界，或者说称得上是绝对主流的方案。
针对云端部署的框架里，我们可以大致分为两类，一种是主要着力于解决推理性能，提高推理速度的框架，这一类里有诸如tensorflow的tensorflow serving、NVIDIA基于他们tensorRt的Triton(原TensorRt Serving)，onnx-runtime，国内的paddle servering等， 将模型转化为某一特定形式（转化的过程中可能伴有一些优化的操作）， 并对外提供服务，以此来获得相对较高的性能。
另一类框架主要着眼于结合模型整个生命周期，对模型部署进行管理，比如mlflow、seldon、bentoml、cortex等等，这些框架的设计与思路其实五花八门，有的为了和训练部分接轨，把模型文件管理也纳入了。有的则是只管到容器编排的部分，用户需要自己做好容器，它帮你发到k8s上之类的（这种情况甚至能和第一类框架连起来用）。当然也有专注于模型推理这一小块的。
写在最后
算法应用落地部署已然成为AI领域关键的一环，由于国外产品制裁，我们也大力支持国产智能硬件AI落地，已在海思、瑞芯微、sigmastar、寒武纪、地平线等国产芯片部署多款算法，如目标检测（YOLOV5等）、人脸识别（scrfd+arcface等）、姿势识别（lite-hrnet等）、动作序列识别（tsm等），目标追踪（MOT,bytetrack）,拥有多行业、多领域真实数据集，并形成多款AI智能产品，落地应用在安防、加油站、充电桩、火车站、商场等各大行业，后续也会开设专题介绍各大智能硬件、各大算法的详细部署流程，致力于发展壮大国产AI部署社区生态。
今天就先到这里，谢谢，点点关注不迷路。
智驱力-科技驱动生产力