Recording and Replaying Field Data with ROS 2 Bags: A Research Workflow

TL;DR: ROS 2 bag recording and replay lets you capture selected topics your robot publishes during a field trial, then repeatably replay the data on a workstation to develop perception, SLAM, or control algorithms offline. Use the rosbag2 CLI with the MCAP storage plugin for large field datasets (chunk compression, fast random access, schema-aware), filter or split bags before replay, and reproduce results with --clock and use_sim_time:=true. This guide shows the full record – filter – replay loop with working commands, suitable for a Leo Rover-class UGV.

Why ROS 2 bag recording and replay matters for field robotics

Field experiments are expensive. Driving a UGV across a quarry, a Mars-analog site, or a forest plot consumes batteries, daylight, and operator time. You rarely get a second shot at the same conditions. ROS 2 bag recording and replay solves this by serializing messages on selected topics to disk, so you can iterate on algorithms back at the lab against the exact data the robot saw.

The workflow matters in three concrete ways. First, it decouples data acquisition from algorithm development – a perception engineer does not need the rover to debug a point cloud filter. Second, it enables regression testing: when you change a node, you can replay the same bag and diff the outputs. Third, it is the foundation of dataset curation for machine learning, where you label frames from a bag and feed them into training pipelines.

The rosbag2 stack in ROS 2 (Humble, Jazzy, Kilted/Rolling) is plugin-based. Two storage backends matter in practice: the historical SQLite3 backend and MCAP, the latter developed by Foxglove and now the recommended format for new projects in recent ROS 2 releases.

What MCAP and SQLite3 actually differ in

Both backends store serialized CDR messages with timestamps and topic metadata. The differences show up when bags get large or when you need to seek inside them.

• SQLite3 (.db3): the historical default. Each bag is a SQLite database. Reliable, but writes can stall under high throughput, and the file format is not optimized for sequential reads of mixed topic streams.
• MCAP (.mcap): a chunked, indexed, append-only container designed for robotics logs. It supports per-chunk compression (LZ4, Zstd), embedded message schemas, and fast time-based seeking. The Foxglove team reports that MCAP typically reads sequentially faster than SQLite3 for large multi-topic bags; benchmark with your own data before committing.

For Leo Rover field campaigns where lidar, stereo cameras, and IMU run simultaneously, MCAP with Zstd chunk compression is the practical default. Install the plugin once:

sudo apt install ros-$ROS_DISTRO-rosbag2-storage-mcap

Create mcap_writer_options.yaml for MCAP chunk compression:

mcap:
  compression: "Zstd"
  compressionLevel: "Fast"

How to record a bag in the field

A typical Leo Rover stack publishes wheel odometry, IMU, camera image, camera info, and optionally a lidar scan. Recording all of it during a 30-minute traverse produces a few gigabytes of data. The basic command is straightforward, but a few flags matter for field use.

Before driving, decide which topics you actually need. Recording /rosout and /parameter_events is rarely useful and inflates the bag. A targeted recording for a navigation experiment looks like this:

ros2 bag record -s mcap --storage-config-file mcap_writer_options.yaml -o quarry_run_01 \
  /tf /tf_static /odom /imu/data /scan /camera/image_raw/compressed /camera/camera_info /cmd_vel

Several flags deserve attention when you record on a robot that may lose power or run out of disk:

• --max-bag-size 2147483648 splits the output into 2 GB chunks, so a crash loses at most the currently open chunk.
• --max-bag-duration 300 rotates files every 5 minutes, useful when you want to discard segments after the fact.
• --compression-mode message with --compression-format zstd compresses per message instead of per file if you use rosbag2’s generic compression rather than MCAP’s chunk compression, trading CPU for finer-grained recovery after interruptions.
• --include-hidden-topics captures topics starting with _; usually you leave this off.
• --qos-profile-overrides-path qos.yaml forces specific QoS on subscriptions, which matters for sensor topics published with BEST_EFFORT reliability.

For a Leo Rover running on a Raspberry Pi, prefer compressed image topics (image_raw/compressed) over raw images. A 1280×720 raw RGB stream at 30 Hz is roughly 80 MB/s; the JPEG-compressed equivalent is one to two orders of magnitude smaller and the Pi’s SD card or USB SSD can keep up.

How to filter and inspect bags before replay

Raw field bags almost always need cleanup. You might want only the segment where the robot was actually moving, or you might want to drop a noisy topic. The ros2 bag CLI provides the tools.

Start by inspecting what you captured:

ros2 bag info quarry_run_01

This prints duration, message counts per topic, and the storage format. To extract a time window or a subset of topics, use the convert verb with a YAML config, or the simpler replay-and-re-record approach. A practical filter that keeps only navigation-relevant topics between two timestamps:

Create filter.yaml:

output_bags:
  - uri: quarry_run_01_filtered
    storage_id: mcap
    topics: [/tf, /tf_static, /odom, /imu/data, /scan]
    start_time_ns: 1700000120000000000
    end_time_ns: 1700000480000000000

Then run:

ros2 bag convert -i quarry_run_01 -o filter.yaml

For programmatic filtering – for example, dropping every other IMU message to downsample – write a short Python script using the rosbag2_py API. The SequentialReader and SequentialWriter classes give you message-level control.

How to replay a bag repeatably

Replay is where most subtle bugs appear. The default ros2 bag play publishes messages at the recorded pace using the player process clock, which is fine for visualization but often insufficient for offline algorithm development. You want the rest of your ROS graph to use the bag’s timestamps, not the system clock.

The canonical replay command for algorithm work is:

ros2 bag play quarry_run_01_filtered --clock 100

The --clock 100 flag publishes /clock at 100 Hz. Every node you launch against the replayed data must be started with use_sim_time:=true:

ros2 run slam_toolbox async_slam_toolbox_node --ros-args -p use_sim_time:=true

Other replay flags that matter in practice:

• --rate 0.5 plays at half speed, useful when your processing pipeline cannot keep up in real time.
• --start-offset 30.0 skips the first 30 seconds, for example to skip the rover boot sequence.
• --loop repeats the bag, handy for stress-testing a node.
• --topics /scan /odom replays only specified topics, which lets you inject live data on the remaining topics.
• --remap /odom:=/odom_recorded renames topics on replay, so you can compare a fresh odometry estimate against the recorded one.

For SLAM and localization development, this loop – record once, replay many times with tweaked parameters – is the core productivity gain. The same workflow is useful when validating Leo Rover navigation stacks before customer deliveries.

How to handle large field datasets

A multi-hour outdoor traverse with lidar and stereo can produce tens or hundreds of gigabytes. A few practices keep this manageable.

• Record to an external SSD over USB 3. SD cards on a Raspberry Pi throttle under sustained write loads above approximately 30 MB/s; a SATA SSD in a USB 3 enclosure typically sustains 200-400 MB/s.
• Use MCAP with Zstd chunk compression. In testing on uncompressed or lightly compressed mixed sensor data, Zstd can often reach 2-4x compression; already JPEG-compressed images may gain much less. Verify on your own data.
• Split bags by time. Combined with --max-bag-duration, this lets you ship only the relevant segments off the robot. Use rsync --partial over the Leo Rover’s WiFi to transfer chunks as they close.
• Strip topics you do not need before archiving. Camera info and TF static are tiny; raw images are not. A filtered archive bag is what goes into long-term storage.
• Generate a manifest. Run ros2 bag info and store the output alongside the bag, together with the URDF, the parameter YAMLs, and the git SHA of the launch repository. Replay without this metadata is guesswork.

For space-analog and planetary exploration scenarios, where reproducibility matters as much as the data itself, bundling the bag with its software environment in a container is standard practice. See how this fits into space robotics workflows built on the Leo Rover platform.

What a minimal end-to-end example looks like

Putting record, filter, and replay together, a single experiment session on a Leo Rover might run as follows. First, on the rover:

ros2 bag record -s mcap --storage-config-file mcap_writer_options.yaml \
  --max-bag-size 1073741824 -o field_session_2024_03_15 \
  /tf /tf_static /odom /imu/data /scan /camera/image_raw/compressed

After the run, transfer the directory and inspect it on the workstation:

ros2 bag info field_session_2024_03_15

Filter to the useful window and topics with a convert YAML as shown above. Then replay against a fresh SLAM stack:

ros2 bag play field_session_2024_03_15_filtered --clock 200 --rate 1.0

In another terminal, launch the SLAM node with use_sim_time:=true, visualize in RViz2 (also with use_sim_time set), and iterate. Each parameter change costs seconds, not a field trip.

FAQ

Q: Should new ROS 2 projects use MCAP or SQLite3?

A: For new projects on ROS 2 Humble and later, MCAP is the practical default, particularly for field datasets with images and lidar. SQLite3 remains the historical default in some distributions and is fine for small bags or compatibility with older tooling. Recent ROS 2 releases move toward MCAP as the recommended format.

Q: How do you avoid dropped messages while recording high-rate sensors?

A: Write to an SSD over USB 3, not an SD card. Use MCAP chunk compression or file-level compression rather than message-level compression when CPU is the bottleneck on the robot. Subscribe with matching QoS by supplying --qos-profile-overrides-path when the publisher uses non-default reliability. Monitor with ros2 topic hz during a dry run.

Q: Why are my replayed timestamps ignored by other nodes?

A: Because those nodes were started without use_sim_time:=true, so they use the system clock. Always launch ros2 bag play with --clock and set use_sim_time on every node, including RViz2, that consumes replayed data.

Q: Can you edit messages inside a bag?

A: Not in place. Use rosbag2_py to read messages from one bag, modify them in Python, and write them to a new bag. This is the standard approach for fixing a wrong frame_id or recalibrating timestamps after the fact.

Q: How do you merge two bags recorded simultaneously on different machines?

A: Use ros2 bag convert with multiple -i input bags and an output YAML, or write a short Python script using two readers and one writer, merging by timestamp. Make sure clocks on both machines were synchronized (chrony or PTP) before the recording.

Q: What is the right way to share a bag with collaborators?

A: Ship the MCAP file, a README with the ros2 bag info output, the URDF, parameter YAMLs, the ROS 2 distribution version, and the git SHA of any custom nodes. Without this metadata, replay results are not reproducible.

Q: Does bag replay work with Nav2 and SLAM Toolbox out of the box?

A: Yes, provided you set use_sim_time:=true on every Nav2 and SLAM Toolbox node and publish /clock from the bag. TF must be present in the bag (or republished), and the URDF must match the robot used during recording.

Put the workflow to work on your robot

The record – filter – replay loop is one of the highest-leverage habits in ROS 2 development. It turns a single field trip into a dataset you can mine for months. If you are evaluating platforms for outdoor experiments, sensor benchmarking, or algorithm research, explore how Leo Rover supports this workflow at Fictionlab’s research applications page.