TensorRT is the most popular inference engine for deploying trained models on NVIDIA GPUs for inference. Not surprisingly, this library and its set of tools are developed by NVIDIA and it is available free for download and use. A new version of TensorRT typically accompanies the release of every new NVIDIA GPU architecture, adding optimizations for the new GPU architecture and also support for new types of layers, operators, and DL frameworks.
NVIDIA TensorRT
Installing TensorRT
TensorRT installers can be downloaded from the web at https://developer.nvidia.com/tensorrt. Installation packages are available for x86-64 (Intel or AMD 64-bit CPU) computers, PowerPC computers, embedded hardware such as NVIDIA TX1/TX2, and NVIDIA...