Computer Vision

Computer Vision – definition

Computer vision is the field of computer science and engineering that extracts structured information from images or image sequences. In mobile robotics, it is used to estimate geometry, motion, scene structure, object presence, terrain properties, and semantic context from camera data. For a UGV, computer vision is not only image processing. It is part of the perception stack that converts raw sensor measurements into data usable by localization, navigation, inspection, and autonomy modules.

In ROS 2-based mobile platforms, computer vision usually operates on streams such as monocular RGB images, stereo pairs, or RGB-D frames. These streams are published with timestamps, frame identifiers, and camera calibration data. The most common ROS message types are sensor_msgs/msg/Image and sensor_msgs/msg/CameraInfo. Camera models and distortion handling follow ROS camera conventions, and frame conventions follow REP 103. In practice, this means that a vision pipeline on a UGV must handle synchronization, camera intrinsics, extrinsics, image transport, and coordinate transforms through tf2.

On platforms such as Leo Rover and Raph Rover, computer vision is typically used for visual odometry, fiducial detection, obstacle segmentation, terrain classification, teleoperation support, and visual inspection. It is not a replacement for a full autonomy stack. Leo Rover supports ROS 2, typically with a Raspberry Pi-based onboard compute unit, so vision workloads must be selected with attention to CPU, memory, and thermal limits. Raph Rover, due to its larger payload capacity, can host more demanding compute and sensor configurations, which is relevant for stereo depth, multi-camera rigs, or GPU-accelerated inference.

How computer vision works in ROS 2

In ROS 2, a vision pipeline is a graph of nodes that subscribe to image topics, process frames, and publish derived outputs. The outputs may include pose estimates, feature tracks, object detections, segmentation masks, point clouds, or occupancy updates. The quality of the result depends on calibration, exposure control, synchronization, and deterministic timing.

A minimal camera integration in ROS 2 usually includes these elements.

Camera driver publishing /image_raw or /image_rect
/camera_info with intrinsics and distortion parameters
Static or dynamic transforms between base_link, camera frame, IMU frame, and other sensors
Optional rectification and compression through image_pipeline and image_transport
Downstream nodes for detection, depth estimation, SLAM, or visual odometry

Typical ROS 2 message types used in computer vision for UGVs are listed below.

Function	ROS 2 message type	Notes
Image stream	`sensor_msgs/msg/Image`	Raw or rectified image data
Camera calibration	`sensor_msgs/msg/CameraInfo`	Intrinsic matrix, distortion model
Point cloud from depth	`sensor_msgs/msg/PointCloud2`	Often generated from stereo or RGB-D
Pose estimate	`geometry_msgs/msg/PoseStamped`	Used by odometry or tag tracking
Odometry	`nav_msgs/msg/Odometry`	Visual odometry or fused state
Transform tree	`tf2_msgs/msg/TFMessage`	Frame relationships per REP 103

The following example shows a simple ROS 2 launch pattern for a USB camera node and an image processing node.

camera_node:
  ros__parameters:
    video_device: /dev/video0
    image_width: 1280
    image_height: 720
    pixel_format: mjpeg
    framerate: 30.0
    camera_frame_id: camera_link

image_proc:
  ros__parameters:
    use_system_default_qos: true

ros2 run tf2_ros static_transform_publisher \
0.12 0.0 0.18 0.0 0.0 0.0 base_link camera_link

Core tasks in mobile robotics

Computer vision on a UGV is usually task-specific. The same camera may support teleoperation, visual localization, and terrain analysis, but each task has different latency and robustness requirements.

Feature detection and tracking – corners, keypoints, descriptors, optical flow
Visual odometry – camera-based motion estimation between frames
Visual SLAM front-end – landmark extraction and loop-closure support
Depth estimation – stereo matching or RGB-D sensing for local mapping
Object detection – cones, pallets, markers, tools, infrastructure elements
Semantic segmentation – drivable area, vegetation, obstacles, terrain classes
Fiducial tracking – AprilTag or ArUco-based pose estimation
Inspection support – crack detection, corrosion indicators, component state

For mobile robots, vision is often fused with IMU, wheel odometry, and LiDAR. A monocular camera alone has scale ambiguity. A stereo or RGB-D camera can recover depth directly, but range quality degrades with distance, low texture, glare, fog, or sunlight. IMU fusion improves short-term motion estimation. LiDAR often provides more reliable obstacle geometry outdoors.

Key parameters and metrics

Vision systems are constrained by optics, compute, bandwidth, and scene conditions. For an engineering evaluation, the relevant metrics should be stated explicitly.

Parameter	Typical value	Why it matters
Resolution	640 x 480 to 1920 x 1080	Higher detail, higher bandwidth and CPU load
Frame rate	15 – 60 Hz	Motion smoothness and odometry stability
Latency	< 100 ms preferred for teleoperation; lower for fast control loops	Affects reactive navigation and teleoperation
Field of view	70 – 120 degrees	Trade-off between coverage and distortion
Calibration error	Application-dependent	Directly impacts pose and depth accuracy
Synchronization offset	As low as possible	Critical for stereo and visual-inertial pipelines

Data rate is also operationally important. Uncompressed RGB8 image throughput can be approximated as:

bandwidth [bytes/s] = width x height x channels x fps

For 1280 x 720, 3 channels, 30 fps, the raw stream is about 82.9 MB/s in decimal units, before transport overhead. In binary units, this is about 79.1 MiB/s. On small onboard computers this has immediate impact on CPU, memory copy cost, and storage.

Supported hardware and integration on Leo Rover and Raph Rover

On Leo Rover, computer vision is usually implemented with a USB camera, stereo camera, or RGB-D sensor integrated as a payload. Since the platform uses a Raspberry Pi-class onboard computer by default, practical pipelines should be lightweight unless additional compute is added. AprilTag detection, basic OpenCV filtering, low-rate teleoperation video, and selected visual odometry pipelines are realistic. Large neural models and dense multi-camera perception often require external acceleration.

On Raph Rover, the larger payload budget makes it more suitable for higher-power computers such as NVIDIA Jetson-class devices, more stable sensor mounts, and combined perception stacks with LiDAR, IMU, GNSS/RTK, and cameras. This matters in field robotics, where camera-only perception is sensitive to dust, low light, repetitive textures, and dynamic shadows.

In both platforms, camera mounting should define:

Extrinsics relative to base_link
Vibration isolation
Known height above ground
Cable routing and power stability
Ingress protection and exposure constraints for outdoor operation

Limitations and trade-offs

Computer vision is powerful, but on UGVs it has well-known failure modes. Performance depends strongly on illumination, blur, texture, weather, and lens cleanliness. A detector validated indoors may fail in outdoor glare. A depth camera that performs well at close range may become unreliable in sunlight or at longer distances. These are engineering constraints, not edge cases.

For this reason, vision is commonly combined with LiDAR, IMU, wheel odometry, or GNSS/RTK. In ROS 2 systems, fusion is often handled downstream in localization and navigation components rather than inside the vision node itself.

Normative references and standards

The technical interpretation of computer vision in ROS-based UGV systems is usually aligned with these sources.

ROS REP 103 – Standard Units of Measure and Coordinate Conventions
ROS 2 documentation for sensor_msgs, image_transport, camera_info_manager, and tf2
OpenCV project documentation for image processing and calibration models
Intel RealSense camera specifications for RGB-D operating ranges and supported stream modes
NVIDIA Jetson platform specifications for accelerated inference workloads
IEEE literature on visual odometry, SLAM front-ends, and perception benchmarking