LiDAR SLAM in Open Terrain: Benchmarking Three SLAM Algorithms on One Rover

TL;DR: This article describes a reproducible approach to comparing slam_toolbox, LIO-SAM, and RTAB-Map on a single Leo Rover platform in mixed outdoor terrain. A meaningful comparison requires the same recorded sensor data, clearly documented configurations, a valid reference trajectory, and publication of the rosbag and launch files. Results such as drift, loop-closure time, CPU load, and RAM use should be reported only for measurements that can be independently reproduced from the published data.

Why does lidar slam outdoor comparison matter for field robots?

Indoor SLAM benchmarks rarely predict outdoor behavior. In open terrain you deal with sparse geometry, vegetation that moves, uneven ground that tilts the sensor, and GNSS that may or may not be reliable under canopy. If you are choosing a stack for a research UGV or a mapping payload on a Leo Rover, you need to know how each algorithm behaves on the same data, not on curated datasets recorded in different conditions.

This article outlines a field-robotics benchmark methodology for three widely used SLAM packages using identical hardware, trajectory, and sensor payload. To make the comparison reproducible, raw measurements, configuration files, software versions, and the rosbag should be published alongside the conclusions.

What hardware and sensor stack was used?

The test platform should be a Leo Rover equipped with a fixed LiDAR and sensor payload. The configuration must remain unchanged throughout the recording session to remove hardware variance from the comparison.

• Platform: Leo Rover, 4×4 skid-steer platform, with wheel odometry available from the rover’s encoders
• Compute: NVIDIA Jetson Orin Nano or equivalent computer running a ROS 2 distribution supported by the selected SLAM packages
• LiDAR: a 3D LiDAR with documented scan rate, field of view, mounting position, and LiDAR-to-base transform
• IMU: an IMU with timestamps synchronized to the LiDAR; LIO-SAM additionally requires correctly configured LiDAR-to-IMU extrinsics
• Camera (optional for RTAB-Map): an RGB or RGB-D camera with calibrated intrinsics and a known camera transform
• Reference trajectory: RTK-GNSS can be used in open sky, but its quality, antenna lever arm, coordinate frame, and degraded accuracy under canopy must be accounted for

A representative route can include paved ground, mowed grass, and a forest path before returning to its start point. The route length, terrain composition, rover speed, weather, and vegetation conditions should be recorded, because each can materially affect SLAM performance.

How were slam_toolbox, LIO-SAM, and RTAB-Map configured?

Each stack should be tuned using documented parameters suitable for outdoor operation, without tuning specifically against the evaluated trajectory. Package versions, forks, commits, launch files, and parameter files must be reported because their defaults and ROS 2 support differ between releases.

slam_toolbox (2D)

A 3D point cloud can be projected to a 2D scan using pointcloud_to_laserscan. The selected vertical band must be expressed in a documented target frame and should exclude the ground, rover body, and as much vegetation as practical. The appropriate band depends on the LiDAR mounting height and local terrain.

Example parameters to document:

• mode: mapping
• resolution: 0.05
• max_laser_range: 60.0
• loop_search_maximum_distance: 5.0

LIO-SAM

LIO-SAM is a lidar-inertial system that requires time-aligned IMU data and correctly configured sensor extrinsics. The original LIO-SAM project targets ROS 1; when using ROS 2, the specific maintained ROS 2 port and its compatibility with the chosen ROS 2 distribution must be identified. GPS fusion can be disabled when evaluating lidar-inertial performance alone, but this should be stated clearly because GNSS fusion changes the nature of the comparison.

RTAB-Map

RTAB-Map can combine visual loop-closure detection with LiDAR scan or scan-cloud registration. A configuration using subscribe_scan_cloud: true can use a 3D LiDAR cloud for ICP, while RGB images and camera calibration can support appearance-based loop closures. RTAB-Map normally uses ORB features by default; SURF requires an OpenCV build with the relevant non-free functionality enabled and should not be assumed to be available in a standard installation.

The recorded dataset may contain the same wheel-odometry and IMU topics for every test, but each package should use only the inputs it supports and that are explicitly enabled in its configuration. In particular, LIO-SAM requires IMU data, whereas slam_toolbox is fundamentally a 2D laser-SLAM package.

What are the benchmark results?

For a fair offline comparison, replay the same rosbag at a rate that every tested stack can process without dropping data. CPU and RAM measurements should include the sampling method, averaging interval, Jetson power mode, and whether GPU utilization is reported separately.

Endpoint error alone is not a complete measure of SLAM drift. If RTK-GNSS is used as a reference, compare trajectories only after verifying timestamp synchronization, coordinate transforms, antenna offset, RTK fix quality, and periods affected by multipath or canopy. Report trajectory metrics such as ATE and RPE in addition to loop-closure behavior.

Results from the benchmark session should be reported in a reproducible form:

• slam_toolbox: report endpoint error or ATE/RPE, CPU and RAM use, loop-closure behavior, and failures caused by an inconsistent 2D scan slice.
• LIO-SAM: report endpoint error or ATE/RPE, CPU and RAM use, loop-closure behavior, and the effect of IMU calibration and rover pitch on the trajectory.
• RTAB-Map: report endpoint error or ATE/RPE, CPU and RAM use, loop-closure behavior, and whether visual conditions affected feature matching.

Behavior by terrain type should also be evaluated separately:

• Paved courtyard: compare short-range tracking and repeatability on stable, geometrically distinct surfaces.
• Mowed grass: check whether vegetation enters the 2D scan band and whether the 3D LiDAR registration remains constrained by stable features.
• Forest path: assess the effect of repeated tree geometry, changing illumination, canopy-related GNSS degradation, and vehicle pitch.

Which SLAM stack fits which use case?

The choice depends on the required map representation, terrain, sensors, and compute budget. For a 2D navigation stack on a low-power UGV operating on mostly flat and firm terrain, slam_toolbox can be a reasonable choice because it works with 2D laser scans and is generally less demanding than 3D lidar-inertial mapping. If you need a lidar-inertial 3D trajectory and can provide calibrated, synchronized IMU data, LIO-SAM or a compatible ROS 2 port may be suitable. If appearance-based loop closure and visual data are useful, RTAB-Map can combine camera information with LiDAR registration.

For teams evaluating these stacks for research deployments, the practical takeaway is that a benchmark on your own data matters more than a result from another site. Publish the rosbag, configurations, package versions, calibration data, and evaluation procedure if you want others to verify the comparison.

How can you reproduce this benchmark?

A reproducible package should include the rosbag, calibration files, launch files, parameter files, exact dependency versions, and an evaluation script. Disk-space requirements depend on the LiDAR model, recording duration, image streams, compression, and generated outputs.

Steps to reproduce:

1. Obtain the benchmark repository and verify that its commit, release artifacts, and software versions are publicly available.
2. Download the recorded rosbag and verify that it contains the required LiDAR, IMU, wheel-odometry, camera, TF, and reference-trajectory topics.
3. Build the workspace using the ROS 2 distribution and dependencies documented by the repository.
4. Launch one SLAM stack at a time with the documented topic remappings, transforms, and parameter file.
5. Record the resulting trajectory and map topics, including /tf, /tf_static, and the map output where applicable.
6. Compare the estimated trajectory with the validated reference trajectory using a documented procedure, for example evo for ATE and RPE.

All launch files should use the same TF tree and topic conventions where the packages allow it. A fourth SLAM package, such as KISS-ICP or FAST-LIO2, will usually require its own sensor-driver assumptions, topic remappings, and calibration validation rather than only minimal edits.

FAQ

Which LiDAR SLAM algorithm is best for outdoor use?

There is no universal answer. The best choice depends on terrain, LiDAR model, IMU quality, camera availability, compute budget, and whether you need 2D or 3D output. Evaluate the candidate stacks on representative recordings from your own operating environment.

Can slam_toolbox work outdoors?

Yes, particularly on flat terrain with stable and geometrically distinct features. It can struggle when a projected 2D laser slice contains moving vegetation or when vehicle pitch causes the slice to no longer represent a consistent plane. Projecting a carefully selected vertical band from a 3D LiDAR can help, but does not eliminate these limitations.

Do you need an IMU for LIO-SAM?

Yes. LIO-SAM is a lidar-inertial system and requires compatible, time-synchronized IMU measurements with correctly configured extrinsics relative to the LiDAR. The exact IMU message requirements and recommended rate depend on the LIO-SAM version or ROS 2 port being used.

How much drift is acceptable for field robotics?

It depends on the task. Agricultural row-following, inspection, and repeated autonomous operation may require sub-meter global accuracy and often benefit from GNSS, landmarks, or other absolute references. For exploration or coarse mapping, larger local error may be acceptable if loop closure or later map optimization corrects the trajectory.

Can you run these SLAM stacks on a Raspberry Pi?

slam_toolbox can run on a Raspberry Pi with a suitable 2D LiDAR, depending on scan rate and the rest of the ROS workload. Real-time 3D lidar-inertial SLAM and camera-assisted RTAB-Map are substantially more demanding; whether they run acceptably on a Raspberry Pi depends on sensor rates, resolution, algorithm settings, and the required update rate.

Does the Leo Rover come with a LiDAR?

LiDAR availability depends on the selected Leo Rover configuration and optional accessories. Check the current product specification and included equipment before purchase.

Where can you get the rosbag and configs?

A reproducible benchmark should provide a public repository containing launch files, parameter files, calibration data, an evaluation script, and a verified download location for the rosbag. Verify that the linked materials are available before relying on any reported benchmark result.

Get the sensor stack used in this benchmark

If you want to run an outdoor SLAM benchmark on your own terrain, check the Fictionlab shop for currently available Leo Rover sensor options. Confirm the included sensors, mounts, wiring, ROS support, and compatibility with your chosen SLAM package before ordering.