模型INT8轉換整理文檔

1、模型INT8轉換整理文檔為評估在GPUT4模型推理性能和資源消耗,需要考慮將模型進行量化后的處理.FP16模型的量化比擬簡單,只需要改動API的一些參數(shù)即可；INT8的實現(xiàn)比擬復雜,特作此整理,以方便以后重復此操作.2 .環(huán)境效勞器：03CPU:48xE5-2650v42.20GHz內存：62GGPUI:TeslaT4顯存：15GTensorRT：CUDA10.1GPU驅動版本：418.116.00CUDNN:7.5.0CUBLAS:10.1模型：googlenet3 .實現(xiàn)方法參考文檔TensorRT/samples/sampleINT8/R

2、EADME.md說明,卞據非Caffe模型轉換方法實現(xiàn),原文局部如下：Generatingbatchfilesfornon-CaffeusersFordevelopersthatarenotusingCaffe,orcannoteasilyconverttoCaffe,thebatchfilescanbegeneratedviathefollowingsequenceofstepsontheinputtrainingdata.Subtractoutthenormalizedmeanfromthedataset.Cropalloftheinputdatatothesamedimensions.S

3、plitthedataintobatchfileswhereeachbatchfilehasNpreprocessedimagesandlabels.GeneratethebatchfilesbasedontheformatspecifiedinBatchfilesforcalibration.4 .操作步驟4.1. 準備校正集文件CalibrationFile編譯命令：g+createCalibrationCache.cpp-lopencv_imgproc-lopencv_core-lopencv_imgcodecs-occc-g執(zhí)行命令：./ccccachelmgs.calibILSVRC

4、2021_val_00001234.JPEG注：1、這里ccc即測試程序,cacheImgs.calib是保存的校正集文件第三個參數(shù)是使用的圖片2、可以屢次使用不同的圖片,增加校正集中圖片的數(shù)量,這里只為測試只采用了一張圖片4.2. 使用校正集進行模型轉換并序列化CudaEngine保存這里使用了TensorRT提供的trtexec樣例程序,執(zhí)行命令：cd/home/reny/downloads/TensorRT-/samples/trtexec./trtexec-deploy=data/googlenet/totxt-output="prob&qu

5、ot;-model=data/googlenet/googlenet.caffemodel-int8-batch=4-saveEngine=googlenet_int8_bs4.engine-calib=CalibratorcreateCache/cacheImgs.calib4.3. 加載CudaEngine文件并反序列化這里采用了TensorRT提供的SampleGoogleNet樣例程序主要改動：增加一個加載CudaEngine文件的函數(shù),該函數(shù)思路源于trtexec樣例代碼/addbyreny2021/12/27staticnvinfer1:ICudaEngine*LoadEngine

6、(conststring&sSavedEngineFile)ICudaEngine*engine;std:vector<char>trtModelStream;size_tsize0;std:ifstreamfile(sSavedEngineFile.data(),std:ios:binary);if(file.good()file.seekg(0,file.end);size=file.tellg();file.seekg(0,file.beg);trtModelStream.resize(size);file.read(trtModelStream.data(),siz

7、e);file.close();)IRuntime*infer=createInferRuntime(gLogger.getTRTLogger();/*if(.useDLACore>=0)infer->setDLACore(gParams.useDLACore);)*/engine=infer->deserializeCudaEngine(trtModelStream.data(),size,nullptr);gLogInfo<<sSavedEngineFile<<"hasbeensuccessfullyloaded."<&l

8、t;std:endl;infer->destroy();returnengine;)修改后測試程序路徑：/home/reny/downloads/TensorRT-/samples/sampleGoogleNet_ry4.4. 執(zhí)行正常步驟進行推理cd/home/reny/downloads/TensorRT./bin/sample_googlenet-loadEngine/home/reny/nvidia/googlenet_int8_bs4.engine5 .實現(xiàn)截圖goqgiian0t_intB_b!t4.ongino_1/bin/s>jnpls_

9、g'=KigLand.-.lladdlEnginst/haiw/rony/mvidItkLUal唱jndwMgQ制InferenLi*也唱山也for5逅1電lietllgoliHfonMT1RTTensorPT叼、量*?審打鈍工呷1具51中事$butisHNk說乎tCuOLMi博I/hcinr-j'renjr/nwidijB/jooglenrtint8wbs4.engineha3beea5uuuit3>3ifiJ】lyloaded_interiigoingon町TftTJTw5<h*RF2號,顰加暈工14bytisJUftkediCiiBLAS工乳工用inpiitbuiferiiib：京嗎M41Kinfflp&fice53H5irfxtr2與18褪&tir*es汨+封已收一coststirw:868mgIRan./bin/sasplejgaoElenetwith；£lInput(a)：dataIJOuLputfG：prccb,原MiPAS5DT力sorffT.5nRMgJWNt*,-loadFnglftp|rgtkEs-art：ftrTans口rtiT.5pwd6.參考資料1代碼TensorRT/s