Multispectral camera

Multispectral camera – definition

A multispectral camera is an imaging sensor that records pixel values in several discrete spectral bands. Unlike an RGB camera, which usually provides three broad visible-light channels, a multispectral device separates the incoming light into additional wavelength ranges, commonly including near-infrared (NIR). Each output band is represented as an image with its own pixel values and metadata; depending on the camera architecture, bands may have separate or shared exposure settings and timestamps. Radiance or reflectance values require appropriate calibration.

In mobile robotics and UGV systems, a multispectral camera is a perception payload used to measure spectral differences that may not be visible in standard colour imagery. Typical applications include crop monitoring, vegetation segmentation, terrain classification, material inspection, and data collection for research. It does not provide autonomous navigation by itself. Its data must be calibrated, spatially aligned, and fused with robot pose estimates before it can be used for mapping or decision-making.

Multispectral imaging differs from hyperspectral imaging. A multispectral sensor captures a limited number of relatively broad bands. A hyperspectral sensor captures many narrower and usually contiguous spectral bands. The exact boundary is not universally standardised, so camera documentation should define the number of bands, centre wavelengths, bandwidths, radiometric output, and optical configuration.

Spectral bands and useful indices

A multispectral camera measures reflected or emitted energy in defined wavelength intervals. The available bands depend on the sensor model and optical filters. For field robotics, visible red, green, blue, red-edge, and NIR bands are common because plants reflect strongly in NIR while chlorophyll absorbs visible red light.

The following parameters should be checked in the manufacturer specification before integrating a camera with a UGV.

Parameter	Why it matters in robotics
Band centre wavelength and bandwidth	Defines which spectral response is measured and whether an index is scientifically comparable with another dataset.
Spatial resolution	Determines ground sampling distance at a given camera height and field of view.
Frame rate and exposure time	Controls motion blur and the maximum practical rover speed during acquisition.
Pixel format and bit depth	Affects dynamic range, data handling, and the suitability of images for quantitative processing; radiometric usefulness also depends on calibration.
Global or rolling shutter	Affects geometric distortion when the rover moves or vibrates.
Calibration method	Determines whether pixel values can be converted from digital numbers to radiance and what additional irradiance or reference measurements are needed to estimate reflectance.

A frequently used vegetation metric is the Normalized Difference Vegetation Index (NDVI). It is normally calculated from NIR and red reflectance measurements:

NDVI = (NIR - Red) / (NIR + Red)

NDVI is dimensionless and normally ranges from -1 to 1 when valid non-negative reflectance values are used. The index is sensitive to illumination, shadows, soil background, saturation, camera spectral response, and calibration procedure. The formula and its remote-sensing interpretation are documented by the United States Geological Survey.

Source: USGS – Landsat Normalized Difference Vegetation Index.

ROS 2 representation and coordinate frames

ROS 2 does not define a dedicated multispectral message type. A driver normally publishes each band as a separate image stream or publishes a multi-channel image when the pixel encoding is supported by the consumer node. Standard image transport uses sensor_msgs/msg/Image. Camera calibration and intrinsic parameters are published with sensor_msgs/msg/CameraInfo.

A practical integration should publish image data with acquisition timestamps from the camera, not timestamps assigned after USB transfer or processing. This is important when the images are associated with odometry, IMU measurements, GNSS positions, or transforms in the ROS 2 tf2 tree.

multispectral_camera:
  ros__parameters:
    frame_id: multispectral_camera_optical_frame
    publish_camera_info: true
    publish_raw_bands: true
    band_topics:
      blue: /multispectral/blue/image_raw
      green: /multispectral/green/image_raw
      red: /multispectral/red/image_raw
      nir: /multispectral/nir/image_raw

For a camera mounted on a Leo Rover or Raph Rover, the optical frame should be connected to the robot body frame through a static or calibrated transform. REP 103 defines ROS coordinate conventions, including right-handed axes. REP 105 defines common mobile-platform frame concepts such as base_link, odom, and map. An optical camera frame convention is typically x right, y down, z forward.

Sources: ROS 2 sensor_msgs/Image, ROS 2 sensor_msgs/CameraInfo, REP 103, REP 105.

Integration on Leo Rover and Raph Rover

Leo Rover is a four-wheel differential-drive platform with an integrated Raspberry Pi compute unit and ROS 2 support, including Humble-based software configurations. A multispectral camera can be connected as a USB, Ethernet, CSI, or vendor-specific payload, provided that the driver, power demand, bandwidth, and physical mounting are validated for the selected configuration. Leo Rover requires a navigation stack and sensor integration for autonomous operation.

For field acquisition, the camera is normally mounted above the ground with a fixed downward or forward-downward orientation. The installation must minimise vibration, protect the lens from dust and water, and keep the camera rigid relative to base_link. Leo Rover has an IP55 enclosure rating, but a payload camera may have different environmental protection and must be assessed separately.

Raph Rover is appropriate when the camera system includes heavier payload elements, such as an industrial enclosure, illumination control, a calibration target holder, an edge computer, or a GNSS/RTK receiver. Its larger payload capacity does not remove the need for frame calibration, time synchronisation, or exposure control.

Calibration, synchronisation, and limitations

Raw pixel values are not automatically reflectance values. For repeatable measurements, the workflow should include radiometric calibration, exposure control, lens correction, band registration, and a reference target with known reflectance. Band registration is critical because small inter-band offsets create false edges in NDVI maps and pixel-level classifiers.

Before operating a rover in a changing outdoor environment, engineers should validate the following points.

All spectral bands are temporally aligned or have known capture offsets.
The transform from base_link to the camera optical frame is measured or calibrated.
Image timestamps are compatible with IMU, wheel odometry, and GNSS/RTK timestamps.
Exposure and gain settings are logged with the captured images.
Calibration targets are included when reflectance comparison between sessions is required.

Multispectral cameras are valuable for semantic perception, but they are usually not a replacement for LiDAR, depth cameras, wheel odometry, or IMU data in navigation. Their performance changes with sunlight angle, clouds, shadows, lens contamination, plant movement, and surface moisture. A robust UGV pipeline therefore separates perception products, such as vegetation masks or spectral maps, from the localisation and safety functions of the rover.

Multispectral camera