From rosbag to Dataset: Building an ML Training Pipeline From Field Recordings

TL;DR: A working rosbag to ML dataset pipeline turns raw ROS 2 field recordings into versioned, split, labeled datasets ready for training. The core steps are: extract time-aligned frames from bag files, anonymize personal data where required, label with a schema you control, split by scene or session (not by frame), and version both data and metadata. This article shows the commands, common failure modes (data leakage between splits, unbalanced lighting), and reproducible scripts you can adapt to recordings from a Leo Rover or any ROS 2 robot.

Why a rosbag to ML dataset pipeline matters for field robotics

Field recordings from mobile robots are messy. You get MCAP or SQLite3 bags with camera frames, IMU, GPS, wheel odometry, and TF, all at different rates. Training a perception model directly on raw topics is not practical. You need a deterministic rosbag to ML dataset pipeline that produces the same output every time you run it on the same input, with clear provenance from sensor sample to training tensor.

The typical use case: a Leo Rover drives a test route with a stereo camera and IMU, generating a 4 GB bag per session. You want to train a terrain classifier or a traversability estimator. The following sections walk through each stage, with commands you can adapt for ROS 2 Humble, Jazzy, or other supported ROS 2 distributions.

How to extract and time-synchronize frames from a ROS 2 bag

Frame extraction in a rosbag to ML dataset pipeline has two responsibilities: decode messages to files, and align sensors that publish at different rates. Cameras usually run at 30 Hz, IMUs at 100-400 Hz, GPS at 1-10 Hz. Naive extraction (dump every message) creates samples without corresponding telemetry or labels.

Use a script based on the rosbags Python library or the rosbag2_py API for reproducible extraction. The example below uses rosbags:

The script reads an MCAP bag and writes PNG frames plus a CSV with matching IMU rows using nearest-timestamp matching within a tolerance window.

• pip install rosbags opencv-python numpy pandas
• Open the bag with rosbags.rosbag2.Reader
• Deserialize sensor_msgs/msg/Image and sensor_msgs/msg/Imu
• For each image timestamp, find the closest IMU sample within 10 ms
• Drop unmatched frames and log the drop rate

Use one timestamp source consistently—typically the message header timestamp when it is trustworthy, otherwise the bag-record timestamp. Record that choice in the dataset metadata.

Example command line usage after saving the script as extract_bag.py:

python extract_bag.py --bag ./session_2024_11_03 --out ./dataset/raw --img-topic /camera/image_raw --imu-topic /imu/data --tol-ms 10

Log the drop rate. If a substantial share of frames lacks a matched IMU sample, your tolerance may be too tight, timestamps may come from different clock sources, or the sensors may not be hardware-synchronized. For approximate synchronization on topics with valid header timestamps, ROS 2 provides message_filters.ApproximateTimeSynchronizer, which can be used in a small node that records synchronized tuples during playback.

What to record in the field to make extraction easier

Recording discipline saves hours later. Practical rules for field sessions:

• Record /tf and /tf_static in every bag if later processing requires coordinate transforms or reprojection
• Record /clock only if you use simulated time
• Split bags every 1-2 GB with --max-bag-size when operationally useful, to limit the impact of a damaged recording file
• Store session context separately from the rosbag-generated metadata.yaml, for example in session_context.yaml: operator, location, weather, lighting, terrain type
• Consider MCAP where the installed ROS 2 storage plugins and downstream tools support it; SQLite3 remains widely supported

How to anonymize personal data before labeling

Field recordings often capture bystanders, license plates, or private property. If your dataset will be shared, published, or processed by third-party labelers, assess privacy obligations before making it available. Under GDPR, personal data may require a lawful basis, appropriate safeguards, and data-subject protections; anonymization can reduce risk but must be effective in practice. Handle this before labeling starts, not after.

A practical approach uses a pretrained face and plate detector as a preprocessing pass. Open-source models, including appropriately trained YOLO-based detectors, can be used for face and plate detection. The script blurs detections in place and writes a per-frame JSON with detection boxes so you can audit detections and perform quality checks for false negatives later.

Store the original bags and any audit records containing identifiable information in encrypted, access-controlled storage. Only anonymized frames should enter an external labeling pipeline where possible. Keep a hash of each processed frame so you can verify integrity across the rest of the pipeline.

How to label and structure the dataset

Labeling tools that read image folders plus JSON manifests work well here. CVAT and FiftyOne support COCO-style datasets, while Label Studio can use COCO through its import and conversion workflows. Pick a schema that maps cleanly to your task:

• Semantic segmentation: PNG masks, one class per pixel value
• Object detection: COCO JSON with bbox and category_id
• Traversability regression: CSV with frame_id and a scalar label

Keep labels separate from images on disk. A layout that scales:

dataset/
  raw/session_2024_11_03/frames/*.png
  raw/session_2024_11_03/telemetry.csv
  labels/session_2024_11_03/annotations.json
  splits/train.txt val.txt test.txt
  metadata/sessions.yaml

How to split train, val, and test without data leakage

This is where most field-robotics datasets fail. If you split randomly by frame, consecutive frames from the same 30 Hz video end up in both train and val. The validation loss looks great, the model fails on new sessions. The rule for a rosbag to ML dataset pipeline: split by session or by geographically disjoint segment, never by frame.

A defensible split procedure:

1. Group frames by session_id; a session may span multiple bag files when recording is split
2. Tag each session with lighting (sunny, overcast, dusk), terrain, and location
3. Stratify where the number of independent sessions permits it: aim for comparable lighting and terrain coverage across splits
4. Assign whole sessions to train, val, or test with a fixed random seed
5. If geographic generalization is a goal, verify that routes in different splits are spatially separated by an appropriate buffer

Ratios such as 70/15/15 or 80/10/10 are common starting points, but the number of independent sessions and the intended deployment conditions matter more than an exact ratio. Save the split as a text file of frame IDs and commit it to version control.

Why unbalanced lighting silently ruins models

If 80% of your training frames were recorded between 10:00 and 14:00 on sunny days, the model may learn bright shadow patterns as a feature. Deployment at 16:00 in overcast weather produces a distribution shift the model was never trained for. Before splitting, plot a histogram of recording hour, cloud cover, and mean frame brightness across sessions. Rebalance by recording more sessions in underrepresented conditions, not by discarding data.

How to version datasets and make the pipeline reproducible

Git handles code. It does not handle 40 GB of frames efficiently. Use a data versioning tool:

• DVC: tracks large files with hashes, integrates with Git, supports S3/GCS/local remotes
• Git LFS: simpler, but storage and bandwidth limits depend on the Git hosting provider
• lakeFS or Pachyderm: heavier, for teams with existing data-lake infrastructure

A DVC-based workflow for a rosbag to ML dataset pipeline:

1. dvc init in your dataset repo
2. dvc add dataset/raw to hash and track the extracted frames
3. dvc add dataset/labels after each labeling round
4. Track split and metadata files with Git or DVC, according to their size and update pattern
5. Commit the .dvc files to Git alongside the extraction scripts
6. Push binaries with dvc push to your remote (S3, MinIO, or an SSH server)

Tag Git commits with dataset versions like v0.3-traversability. Anyone can then git checkout v0.3-traversability && dvc pull and retrieve the corresponding training set. Combine this with pinned dependencies, fixed split seeds, recorded preprocessing-model weights, and a documented runtime environment to make results substantially more reproducible across machines.

What a full pipeline run looks like end to end

A typical run against a fresh set of recordings from a Leo Rover expedition:

1. ros2 bag info session_2024_11_03 to verify topics and duration
2. python extract_bag.py --bag session_2024_11_03 --out dataset/raw/session_2024_11_03 --img-topic /camera/image_raw --imu-topic /imu/data --tol-ms 10
3. python anonymize.py --in dataset/raw/session_2024_11_03/frames --model path/to/face-plate-detector.pt
4. Import frames into CVAT, label, export COCO JSON to dataset/labels/
5. python make_splits.py --sessions metadata/sessions.yaml --seed 42 --out dataset/splits/
6. dvc add dataset/raw dataset/labels && git add . && git commit -m "dataset v0.4" && dvc push
7. Train, evaluate, and record the Git commit and DVC-tracked dataset revision in your experiment tracker

For teams running similar pipelines on Fictionlab platforms, the research applications page lists reference configurations for sensor payloads and data-collection workflows.

Frequently asked questions

What bag format should you use for ML data collection in ROS 2?

MCAP is often a good default for new ROS 2 data-collection workflows when the MCAP storage plugin is available in your ROS 2 installation and supported by your tools. It provides chunked storage and indexing features useful for large recordings. SQLite3 remains broadly supported, so validate your recording, extraction, and playback toolchain before standardizing on either format.

How do you handle sensors with different publish rates?

Use nearest-timestamp matching with a tolerance window for offline extraction, or message_filters.ApproximateTimeSynchronizer for online synchronization of messages with valid header timestamps. Log the drop rate and investigate substantial mismatch rates, especially when sensors use different clock sources.

Can you split randomly if sessions are short?

No. Even in a 30-second clip, adjacent frames share content. Split by session, geographic segment, or time block. Random per-frame splits produce optimistic validation metrics that may not survive deployment.

Do you need to anonymize data used only internally?

It depends on the jurisdiction, the purpose of processing, the categories of data, and internal policy. Under GDPR, internal processing of personal data can still fall under the regulation. Assess the processing basis and safeguards early; effective anonymization may simplify later sharing and publication.

Is DVC necessary, or can Git LFS handle a robotics dataset?

Git LFS can work for smaller datasets, but provider storage and bandwidth quotas can become restrictive. DVC supports arbitrary remotes and is often more suitable for large or frequently revised datasets. The choice should reflect dataset size, team workflow, hosting limits, and retention requirements.

How much data does a useful field dataset need?

Task-dependent. The required number of labeled frames depends on model architecture, label quality, class balance, terrain variability, and the target deployment domain. Prioritize diversity of conditions, independent sessions, and representative evaluation data over raw frame count.

Can this pipeline run on the robot itself?

Extraction and anonymization can run on the robot’s onboard computer for small datasets if compute, storage, and power budgets permit, but labeling and training are usually offloaded to a workstation or cloud. On-board recording is generally the primary requirement; benchmark preliminary processing on the specific hardware and sensor configuration.

Put the pipeline to work on your own recordings

A reproducible rosbag to ML dataset pipeline is the foundation for serious perception or learning work on mobile robots. If you are building research workflows around field data collection, explore Fictionlab’s research applications to see how Leo Rover platforms are used in university and industrial labs for these tasks.