Semantic segmentation

Semantic segmentation – definition

Semantic segmentation is a computer vision task that assigns a semantic class label to every pixel of an image. In mobile robotics, the result is a dense scene interpretation rather than a set of object bounding boxes. A segmentation model can label pixels as ground, grass, road, wall, person, vegetation, obstacle, sky, or another class defined by the training dataset.

For an unmanned ground vehicle (UGV), semantic segmentation converts camera data into information that can support traversability analysis, obstacle detection, terrain classification, inspection, and map annotation. It does not directly provide robot position, path planning, or collision avoidance. Those functions require additional components such as state estimation, SLAM, Nav2 costmaps, and a validated control pipeline.

Let an input image have width W, height H, and K semantic classes. A segmentation network produces a label map:

L(x, y) ∈ {0, 1, ..., K - 1},  where 0 ≤ x < W and 0 ≤ y < H

Many networks first produce per-class logits or probabilities. The final class at pixel (x, y) is commonly selected using the maximum predicted probability:

L(x, y) = argmaxk P(class = k | x, y)

Semantic segmentation in UGV perception

A UGV uses semantic segmentation to distinguish meaningful surface and obstacle categories in its local environment. This is particularly useful where geometric sensing alone is insufficient. A LiDAR can detect a height discontinuity, for example, but it cannot inherently distinguish compacted soil from low vegetation or a painted safety zone.

Typical UGV outputs include the following data products. Their class definitions must be documented with the model version and training data.

Class-ID mask – a single-channel image in which each pixel stores an integer class identifier.
Colourised mask – an rgb8 image intended for operator visualisation, not for downstream classification.
Confidence map – per-pixel confidence, probability, entropy, or margin between the two highest-scoring classes.
Traversability mask – a derived binary or weighted representation used to identify terrain that may be suitable for driving.
Projected point cloud labels – semantic labels transferred from calibrated camera pixels to LiDAR points or local map cells.

Semantic segmentation differs from instance segmentation. Semantic segmentation labels all pixels of the same category with the same class, such as all vegetation pixels. Instance segmentation additionally separates individual objects, such as two separate people or two separate pallets. Panoptic segmentation combines semantic “stuff” classes and countable object instances.

ROS 2 data flow and message types

ROS 2 does not define a single core message type dedicated to semantic segmentation masks. In practice, a class mask is commonly published as sensor_msgs/msg/Image, with the image encoding and class mapping documented by the package that publishes it. The ROS image message contains a header, image dimensions, encoding, endianness, row step, and byte data. Its header timestamp should represent the sensor acquisition time where available.

The sensor_msgs/Image specification is maintained in the ROS 2 Common Interfaces documentation. The coordinate conventions used when projecting a mask into 3D should follow REP 103. Camera and robot frame relationships should follow REP 105.

# Inspect an incoming semantic mask
ros2 topic echo /semantic_segmentation/mask --once
ros2 topic hz /semantic_segmentation/mask
ros2 topic info /semantic_segmentation/mask -v

A practical ROS 2 graph normally synchronises the source image, camera calibration, and inference output. If labels are projected into a point cloud, the system also requires a valid transform between the camera optical frame and the LiDAR or base frame at the relevant timestamp.

semantic_segmentation_node:
  ros__parameters:
    image_topic: /camera/color/image_raw
    camera_info_topic: /camera/color/camera_info
    mask_topic: /semantic_segmentation/mask
    confidence_topic: /semantic_segmentation/confidence
    input_width: 640
    input_height: 384
    class_map: config/classes.yaml

The input dimensions above are model configuration values, not universal requirements. Resizing an image before inference changes the spatial relation between source pixels and output pixels. A node should preserve or explicitly publish the scale and crop transformation when another component needs alignment with the original camera image.

Key metrics and validation criteria

Segmentation quality must be evaluated on data that represents the intended operating environment. A model trained on urban road imagery can perform poorly on indoor laboratory floors, agricultural soil, gravel, or unstructured construction terrain. Validation should include the camera position, lighting, weather, motion blur, and terrain types expected by the UGV.

Common metrics are listed below. They should be reported per class because aggregate metrics can hide weak performance on safety-relevant categories.

Intersection over Union (IoU): IoU = TP / (TP + FP + FN).
Mean IoU (mIoU): the arithmetic mean of IoU values over the evaluated classes.
Pixel accuracy: correctly classified pixels divided by all evaluated pixels. It can be misleading when one class dominates the image.
Inference latency: time from receiving an image to publishing the result.
End-to-end age: current time minus source-image timestamp, provided the timestamps use the same clock domain. This is more relevant than inference time alone for a moving rover.
Class confusion: false classification between operationally different categories, such as drivable ground and obstacle.

For navigation, a mask should not be treated as ground truth. Conservative costmap updates, confidence thresholds, temporal filtering, and independent geometric checks reduce the impact of false drivable-area predictions.

Use with Leo Rover and Raph Rover

On Leo Rover, semantic segmentation is typically deployed as an external perception workload connected to the rover’s ROS 2 system. On Leo Rover, ROS 2 runs on the Raspberry Pi-based onboard computer, while the LeoCore controller board handles low-level motor and sensor control. The supported ROS 2 distribution depends on the installed LeoOS image. Model size, image resolution, accelerator availability, and thermal limits determine whether inference can run on the onboard computer or should be offloaded to a companion computer.

Leo Rover has a four-wheel skid-steer drive, so a perception pipeline can use the segmentation output to modify local navigation costs or to stop before a detected non-traversable region. It remains necessary to integrate localization, transforms, sensor drivers, and a navigation stack. Semantic segmentation alone does not make Leo Rover autonomous.

Raph Rover can carry a larger sensor and compute payload where the application requires it. Typical configurations may combine a forward camera with LiDAR and IMU data. Camera labels can classify terrain, while LiDAR supplies range geometry and the IMU supports motion estimation. Sensor calibration is essential: an incorrect camera-to-LiDAR transform can assign a valid pixel label to the wrong 3D point.

Limitations and engineering trade-offs

Segmentation models are sensitive to domain shift. Shadows, lens contamination, rain, low illumination, seasonal vegetation changes, and camera exposure can alter predictions without changing the physical drivability of terrain. A model also cannot infer geometry hidden by occlusion.

Class definitions are another source of ambiguity. “Obstacle” may include a curb, tall grass, loose material, a person, or a negative obstacle, but these categories have different operational consequences. The training label policy must define them precisely. For field robotics, negative obstacles such as holes or drop-offs require particular care because a forward RGB image may provide weak or delayed evidence.

Normative references and technical sources

The following sources define the ROS interfaces and coordinate conventions most relevant to integrating semantic segmentation on a ROS 2 UGV.

ROS 2 sensor_msgs/Image message definition – image transport data structure and fields.
ROS REP 103 – standard units, coordinate conventions, and camera optical-frame orientation.
ROS REP 105 – coordinate frames for mobile platforms, including map, odom, and base_link.
ROS 2 Humble Hawksbill documentation – ROS 2 APIs, tools, and package documentation.
Leo Rover technical documentation – platform-specific ROS 2 integration and hardware documentation.