Object detection

Object detection – definition

Object detection is a computer vision task that identifies objects of predefined classes in sensor data and estimates their image location. For a 2D camera image, a detector usually returns a class label, a confidence score, and a bounding box defined by its centre coordinates, width, and height in pixels.

In mobile robotics, object detection is used to convert camera frames into semantic observations. A UGV can use these observations to identify people, vehicles, tools, crop rows, safety markers, or inspection targets. Detection alone does not provide an obstacle position in the robot coordinate frame. A 2D bounding box must be combined with depth, stereo disparity, LiDAR data, or a calibrated camera model to estimate a 3D position.

Object detection differs from image classification, which assigns one label to a whole image, and semantic segmentation, which assigns a class to individual pixels. Instance segmentation extends detection by producing a separate pixel mask for each detected object.

Object detection outputs in ROS 2

ROS 2 nodes commonly receive images through sensor_msgs/msg/Image and camera calibration data through sensor_msgs/msg/CameraInfo. Image timestamps are needed for temporal alignment, while valid frame IDs allow observations to be transformed into the robot frame.

The vision_msgs package defines interoperable message types for vision pipelines. For 2D detection, the relevant message is vision_msgs/msg/Detection2DArray. Each Detection2D includes a 2D bounding box and one or more classification hypotheses, represented as object hypotheses with optional pose information.

A typical ROS 2 pipeline contains the following data flow:

A camera driver publishes /camera/image_raw and /camera/camera_info.
An inference node subscribes to the image topic and runs a trained model.
The node publishes detections, for example on /detections.
A fusion or tracking node associates detections with depth, LiDAR clusters, or consecutive frames.
A navigation or mission node consumes validated semantic observations.

Image topics should use timestamps from the acquisition device where available. A detector that publishes an output after variable GPU or CPU inference time should preserve the source image timestamp rather than replace it with the publication time.

# Inspect the image metadata and detection interface
ros2 topic echo /camera/camera_info --once
ros2 interface show vision_msgs/msg/Detection2DArray
ros2 topic hz /detections

Coordinate frames and 3D localisation

A bounding box is expressed in image coordinates, not in metres. To use a detection for UGV navigation, the system must determine its location relative to a frame such as base_link, odom, or map. ROS coordinate-frame conventions are defined by REP 103, while the relationship between mobile-robot frames is described by REP 105.

For a calibrated pinhole camera, a pixel coordinate (u, v) and depth Z can be back-projected using the camera intrinsic matrix:

X = (u - cx) × Z / fx
Y = (v - cy) × Z / fy
Z = depth

Here, fx and fy are focal lengths in pixels, while cx and cy are the principal point coordinates. The resulting point is initially in the camera optical frame. A tf2 transform is then required to express it in base_link or another target frame. The static transform must represent the measured camera mounting position and orientation.

Key metrics for a detection pipeline

Detection quality must be evaluated separately from runtime behaviour. A model can achieve good offline accuracy while still being unsuitable for a moving platform if its latency, synchronization, or false-positive rate is not controlled.

Metric	Meaning in a UGV system
Intersection over Union (IoU)	Overlap between predicted and reference bounding boxes: `IoU = area(intersection) / area(union)`.
Precision	Proportion of reported detections that are correct. Low precision produces false targets or false alarms.
Recall	Proportion of relevant objects that are detected. Low recall can cause missed inspection targets or obstacles.
Average Precision (AP)	Area under the precision-recall curve for one class and a specified IoU criterion.
End-to-end latency	Time from image capture to a usable detection message. It affects the spatial error of a moving rover.
Detection rate	Published detection messages per second. It must be assessed together with latency and dropped frames.

The COCO evaluation protocol is a widely used benchmark methodology for AP and IoU-based detection evaluation. Its definitions and reference API are maintained in the COCO detection evaluation documentation. A deployment dataset should represent the real operating environment, including terrain, illumination, camera vibration, motion blur, and target scale.

Integration on Leo Rover and Raph Rover

Leo Rover can run ROS 2 on its built-in Raspberry Pi compute unit and uses differential drive. Object detection therefore requires a camera payload, a ROS 2 camera driver, a detection node, and a defined transform between the camera and rover frames. Leo Rover is not autonomous out of the box. Detection must be integrated with perception, localisation, and navigation logic before it can affect rover motion.

For moderate models, inference can run on the onboard compute unit if measured latency and thermal behaviour are acceptable. More demanding models may use an external accelerator or a separate companion computer. The companion computer must share time, network connectivity, and ROS 2 domain configuration with the rover. Platform-specific integration information should be checked against the Leo Rover technical documentation.

Raph Rover can accommodate applications that require a larger payload, including protected sensor assemblies or higher-power compute hardware. Its larger payload capability does not eliminate the need for calibration, transform validation, or field testing. The perception stack remains responsible for rejecting uncertain detections before they are used by mission software.

# Example detector parameters.
# Names are package-specific and are not a ROS 2 standard.
detector_node:
  ros__parameters:
    image_topic: /camera/image_raw
    camera_info_topic: /camera/camera_info
    detections_topic: /detections
    confidence_threshold: 0.60
    target_frame: base_link
    use_sim_time: false

Limitations and engineering considerations

Object detection is probabilistic. Confidence scores are model outputs, not physical measurements of certainty. A threshold should be selected from validation data and safety requirements, not copied from a generic example.

Common failure sources include poor lighting, occlusion, dust on the camera lens, motion blur, domain shift between training and deployment images, and inaccurate camera calibration. Depth-camera measurements can fail on reflective, transparent, distant, or strongly sunlit surfaces. LiDAR association can improve range estimation, but it requires extrinsic calibration and timestamp alignment between sensors.

For navigation-related use cases, a robust design usually adds temporal tracking, range validation, and explicit rules for stale data. A rover should not treat every single-frame detection as a persistent object or as an immediate navigation constraint.