TensorRT

TensorRT – definition

TensorRT is an NVIDIA software development kit for optimizing and executing trained neural networks on NVIDIA GPUs and NVIDIA Jetson systems. It converts a model from an interchange or framework representation, commonly ONNX, into an optimized inference engine. The engine contains a hardware-specific execution plan that selects kernels, tensor layouts, precision, memory allocation strategy, and layer fusions for the target device.

In mobile robotics, TensorRT is used for perception workloads that must run with bounded latency on an onboard computer. Typical UGV tasks include object detection, semantic segmentation, depth estimation, visual odometry components, traversability estimation, and camera-based inspection. TensorRT is an inference runtime. It does not train models, provide SLAM, publish ROS 2 topics, or control the drivetrain.

TensorRT requires supported NVIDIA hardware and software, typically including an NVIDIA CUDA-capable GPU. Therefore, it cannot execute directly on the default Raspberry Pi compute unit of Leo Rover. A Leo Rover can use TensorRT when an NVIDIA Jetson computer is added as an external or integrated compute node. Raph Rover can host a Jetson-based payload where payload capacity, power availability, and thermal design allow it. The applicable hardware and software compatibility matrix must be checked against the installed NVIDIA JetPack, CUDA, TensorRT, and Ubuntu versions.

Source: NVIDIA TensorRT Documentation; NVIDIA TensorRT Inference Library.

How TensorRT works on a ROS 2 UGV

A ROS 2 perception node receives sensor data, prepares tensors, runs TensorRT inference, and publishes results. TensorRT is normally located inside the inference node rather than exposed as a separate ROS 2 subsystem. This keeps model-specific preprocessing, postprocessing, and engine lifecycle management within one component.

For a camera pipeline, the relevant ROS 2 interfaces commonly include the following messages. Their use depends on the perception package and does not constitute a TensorRT requirement.

Pipeline stage	Typical ROS 2 interface	Purpose
Image input	`sensor_msgs/msg/Image`	Transfers decoded image data and timestamp.
Camera calibration	`sensor_msgs/msg/CameraInfo`	Provides intrinsic parameters and distortion model.
Detection output	`vision_msgs/msg/Detection2DArray`	Represents 2D detections with hypotheses and bounding boxes.
Segmentation output	`sensor_msgs/msg/Image`	Can carry a class-ID mask or colourized segmentation image.

The input image timestamp should be propagated to the output message. This matters when detections are synchronized with LiDAR, IMU, wheel odometry, or TF2 transforms. A perception result generated from an old frame can be geometrically incorrect if the UGV has moved before the result is consumed.

Source: ROS 2 sensor_msgs documentation; ROS 2 vision_msgs documentation; ROS 2 Quality of Service settings.

TensorRT optimization stages

The standard deployment path starts with a trained model and ends with a serialized TensorRT engine. ONNX is often used because it separates model export from the runtime implementation. Unsupported operators, dynamic control flow, custom layers, and preprocessing outside the graph may require TensorRT plugins or application code.

A practical workflow contains these stages:

Export the trained network to ONNX with known input names, shapes, and operators.
Validate numerical output with representative images or recorded ROS 2 data.
Build an engine on the target NVIDIA platform.
Select permitted precisions such as FP32, FP16, or INT8.
Run inference from a ROS 2 node and measure end-to-end latency.

# Build an FP16 engine from an ONNX model on the target Jetson device
trtexec \
  --onnx=terrain_segmentation.onnx \
  --saveEngine=terrain_segmentation_fp16.engine \
  --fp16 \
  --shapes=images:1x3x640x640

TensorRT supports FP32, FP16, INT8, and, on supported hardware and software configurations, additional lower-precision formats. INT8 inference requires calibration using representative data or a model prepared with quantization-aware training and must be validated against the task metric. For example, a segmentation model should be assessed using metrics such as mean intersection over union, not only inference speed.

Engine files are not generally portable across all NVIDIA GPU architectures, TensorRT releases, or build configurations. NVIDIA recommends building engines for the deployment environment. An engine should therefore be treated as a generated deployment artifact, not as a universal model file.

Source: NVIDIA TensorRT ONNX workflow; NVIDIA TensorRT accuracy considerations.

Key latency and throughput metrics

Robotics integration should measure the full pipeline rather than only TensorRT execution time. A detector may report low GPU inference latency while still producing delayed results because of camera capture, image conversion, CPU preprocessing, queueing, or postprocessing.

The main timing quantities can be expressed as:

T_end_to_end = T_capture + T_transport + T_preprocess +
               T_inference + T_postprocess + T_publish

throughput [Hz] = processed or published messages / measurement time [s]

For a serial, non-pipelined pipeline, the maximum throughput is approximately 1 / T_end_to_end. In pipelined systems, throughput and end-to-end latency must be measured separately.

For a Leo Rover or Raph Rover navigation pipeline, record at least the following values from ROS 2 timestamps and profiling tools:

input image resolution and encoding, for example 640x480 or 1280x720;
model input tensor shape, such as Nx3xHxW for an RGB network;
median, 95th percentile, and maximum end-to-end latency;
GPU memory use and CPU memory use;
message drop count and queue depth;
model-quality metrics relevant to the task, such as precision, recall, mean average precision, or mean IoU.

For mobile navigation, latency must be evaluated together with vehicle speed, braking behaviour, sensor field of view, and local planner update rate. TensorRT does not provide these safety or navigation properties.

Deployment considerations for Leo Rover and Raph Rover

Leo Rover can run ROS 2, with supported distributions depending on the installed LeoOS image and software configuration, and uses differential drive. TensorRT-based perception should therefore run on an attached NVIDIA compute platform or on a separate networked ROS 2 computer. The default Raspberry Pi remains suitable for core platform communication and selected ROS 2 nodes, but it does not provide the CUDA execution environment required by TensorRT.

Raph Rover may be appropriate when the sensor suite and compute payload require more space, power, or mass capacity. Typical integrations can combine a Jetson computer with an RGB or depth camera, LiDAR, IMU, and GNSS/RTK receiver. The computer should publish transforms through TF2 and use consistent timestamps across sensor drivers.

tensor_rt_detector:
  ros__parameters:
    engine_path: /opt/models/detector_fp16.engine
    image_topic: /camera/color/image_raw
    camera_info_topic: /camera/color/camera_info
    detections_topic: /perception/detections
    input_width: 640
    input_height: 640
    confidence_threshold: 0.50

Before field deployment, test the system with recorded ROS 2 bags, then validate it on the target UGV under expected lighting, vibration, temperature, and network conditions. Camera exposure changes, motion blur, dust, and terrain-induced vibration can reduce model accuracy even when TensorRT performance is stable.