Depth camera

Depth camera – definition

A depth camera is an imaging sensor that estimates the distance from the sensor to visible scene points and outputs a depth map. In mobile robotics, this map is typically aligned to a 2D image grid, where each pixel stores depth in meters or millimeters, depending on the encoding. In ROS and ROS 2 systems, depth data is usually published as an image topic and optionally converted into a 3D point cloud for perception, mapping, and navigation.

On UGV platforms, a depth camera is used for near-field obstacle detection, local environment reconstruction, visual odometry support, and semantic perception. It does not replace all other sensors. Compared with LiDAR, a depth camera usually provides dense local geometry but with shorter range, lower performance in sunlight for some technologies, and stronger dependence on surface properties. On platforms such as Leo Rover and Raph Rover, depth cameras are most useful as front-facing sensors for indoor navigation, lab research, teleoperation assistance, and short-range terrain perception.

In ROS terminology, depth cameras are commonly represented through image and camera calibration interfaces defined by sensor_msgs/msg/Image, sensor_msgs/msg/CameraInfo, and often sensor_msgs/msg/PointCloud2. Frame naming and coordinate conventions should follow REP 103 and REP 105. This matters when depth data is fused with IMU, wheel odometry, LiDAR, or GNSS on a UGV.

How a depth camera measures distance

Depth cameras used in robotics are not all based on the same physical principle. The integration method, noise profile, and operating envelope depend on the sensing technology. This directly affects SLAM and navigation performance on a mobile base.

The main categories are:

Stereo depth cameras – estimate disparity between two or more synchronized cameras. Depth is derived from triangulation.
Structured light cameras – project a known infrared pattern and infer depth from deformation of that pattern.
Time-of-flight cameras – estimate distance from the travel time or phase shift of emitted light.

For stereo systems, the standard relation is:

Z = (f * B) / d

where Z is depth, f is focal length, B is baseline, and d is disparity. This means depth uncertainty increases with distance, and also with poor texture or calibration error. For UGV use, stereo works well when there is enough visual structure in the scene. It may degrade on uniform walls, glossy surfaces, or low-light terrain.

ROS 2 interfaces and message types

In ROS 2, a depth camera driver usually publishes several synchronized streams. For navigation and mapping, correct time stamping, calibration, and TF are more important than raw image availability alone.

Common ROS 2 interfaces include:

/camera/depth/image_rect_raw – depth image, often encoded as 16UC1 or 32FC1
/camera/depth/camera_info – intrinsic calibration and projection parameters
/camera/color/image_raw – RGB image for visual perception
/camera/color/camera_info – RGB camera intrinsics
/camera/depth/points – organized or unorganized point cloud as sensor_msgs/msg/PointCloud2
/tf and /tf_static – transform tree linking the camera frame to base_link, odom, and map

According to REP 103, ROS uses SI units, but depth image encoding is driver-specific, so values must be interpreted according to the documented message encoding and driver conventions. According to REP 105, the robot should maintain a consistent relation between map, odom, and base_link. A depth camera mounted on a UGV should therefore have a clearly defined optical frame, usually following ROS optical frame conventions.

ros2 topic list
ros2 topic echo /camera/depth/camera_info
ros2 topic hz /camera/depth/image_rect_raw
ros2 run tf2_ros tf2_echo base_link camera_depth_optical_frame

Key parameters and metrics

When selecting a depth camera for a mobile robot, the important question is not only whether it outputs depth, but whether its output is stable and useful for the required operating range. On Leo Rover and Raph Rover, the relevant constraints are compute budget, mounting height, field of view, and environmental conditions.

Parameter	Why it matters on a UGV	Typical unit
Depth range	Defines usable obstacle detection distance	m
Field of view	Affects coverage in front of the robot and side visibility	degrees
Resolution	Controls spatial detail and compute load	pixels
Frame rate	Influences latency and local planner responsiveness	Hz
Depth accuracy	Determines map quality and obstacle inflation reliability	mm or % of distance
Minimum depth	Important for close obstacle sensing near bumpers or payloads	m
Outdoor robustness	Critical in sunlight and high-contrast terrain	qualitative, vendor-specified
Synchronization	Needed for fusion with IMU or wheel odometry	timestamp quality

Manufacturers such as Intel RealSense publish these values in product datasheets and SDK documentation. For engineering work, vendor specifications should be checked against the exact model, resolution, and preset because operating range and precision often vary by mode.

Use in SLAM, navigation, and perception

Depth cameras are widely used in visual SLAM and RGB-D SLAM. In ROS 2, they can feed packages for visual odometry, 3D mapping, obstacle layer generation, or semantic segmentation. They are especially useful in confined spaces where a 2D LiDAR cannot fully capture vertical structure.

Typical uses on UGVs include:

building a local voxel or occupancy map for obstacle avoidance
estimating motion with RGB-D visual odometry
detecting negative obstacles, edges, and low obstacles that may be missed by a high-mounted 2D LiDAR
supporting object detection with aligned RGB and depth
creating point clouds for inspection or research datasets

In Nav2-based ROS 2 stacks, depth data is often integrated through a voxel layer or obstacle layer after conversion to a point cloud. This improves local planning but increases CPU and memory use. On embedded computers, resolution and publish rate usually need to be reduced.

Integration on Leo Rover and Raph Rover

Leo Rover is a compact differential-drive platform with a Raspberry Pi-based compute unit and ROS 2 support. A depth camera can be mounted as a front payload for indoor experiments, education, and near-range autonomy research. Because Leo Rover is not an autonomous platform out of the box, a complete navigation stack still requires calibration, TF configuration, odometry validation, and costmap tuning.

For Leo Rover, practical constraints are:

limited onboard compute compared with x86 or Jetson-class systems
USB bandwidth and power budgeting for camera operation
reduced outdoor performance for some structured-light or active IR sensors
need to tune frame rate and resolution to keep latency acceptable

Raph Rover is larger and intended for higher payload applications. It can carry heavier compute and sensor sets, so a depth camera is more often combined with LiDAR, IMU, GNSS/RTK, and additional cameras. In that setup, the depth camera usually covers the near field, while LiDAR or GNSS supports broader localization and outdoor operation.

obstacle_layer:
  plugin: "nav2_costmap_2d::ObstacleLayer"
  observation_sources: depth_points
  depth_points:
    topic: /camera/depth/points
    data_type: "PointCloud2"
    marking: true
    clearing: true
    obstacle_range: 3.0
    raytrace_range: 4.0
    max_obstacle_height: 1.5
    min_obstacle_height: 0.02

Limitations and trade-offs

A depth camera is not a universal substitute for LiDAR or radar. Its output quality depends on lighting, reflectivity, texture, motion blur, and contamination of the optical path. On outdoor UGVs, direct sunlight can reduce the reliability of active IR systems. Dust, rain, and vibration also degrade measurements.

The main trade-offs are:

dense short-range geometry versus limited range
lower hardware cost versus higher sensitivity to scene conditions
rich RGB-D perception versus higher compute load and calibration complexity
good vertical obstacle capture versus possible failure on transparent or reflective surfaces

Normative references and standards

The ROS coordinate and unit conventions relevant to depth camera integration are primarily defined in REP 103 – Standard Units of Measure and Coordinate Conventions, and REP 105 – Coordinate Frames for Mobile Platforms. ROS image transport and camera calibration are implemented through the standard camera pipeline and sensor_msgs interfaces documented on docs.ros.org. For hardware-level parameters such as depth range, frame rate, and accuracy, the authoritative source is the manufacturer datasheet and SDK documentation for the specific model.

In research papers, depth sensing and RGB-D perception are often described under IEEE robotics and computer vision terminology. For implementation on a UGV, however, the decisive references remain the ROS interface specification, the TF frame design, and the actual sensor specification sheet.