How to Run a Repeatable Outdoor Field Test: A Protocol You Can Cite

TL;DR: A repeatable robot field test protocol defines the route, initial conditions, number of runs, measured quantities, logging method, and rejection criteria before the robot moves. This article gives you a citable protocol you can drop into a paper or report, plus a downloadable result-sheet template. It targets small outdoor UGVs such as Leo Rover and Raph Rover running ROS 2, but the structure applies to any mobile robot outdoors.

Why a robot field test protocol matters

Outdoor experiments are noisy. GNSS drifts, tire pressure changes, grass grows, batteries age, and the wind pushes lightweight platforms off course. Without a written robot field test protocol, two runs are rarely comparable, and a reviewer cannot reproduce your result. The point of a protocol is not bureaucracy. It is to fix every variable you can control so the ones you cannot become measurable.

Field robotics papers sometimes report a single successful traverse or a “typical run.” That is not sufficient evidence on its own. Evidence is a defined route, executed N times under logged conditions, with pre-declared rejection rules and raw data attached. This is good practice in autonomous vehicle benchmarking, and it scales down cleanly to a small rover on a university lawn.

What a repeatable field test actually is

A repeatable field test is an experiment where the same team can execute the same procedure under the same stated conditions and obtain results within a stated tolerance. Repeatability is not reproducibility: repeatability concerns repeated measurements under the same operating conditions, while reproducibility concerns results obtained under changed conditions, such as by another team, at another site, or with another setup. Start with the first, then aim for the second.

Three properties define a repeatable field test:

• Determinism of setup: hardware state, software versions, and environmental bounds are recorded before the first run.
• Statistical validity: enough runs to estimate uncertainty and identify anomalous observations using a pre-declared method.
• Traceable data: raw logs (ROS 2 bags, GNSS traces, IMU) stored with a checksum and a run ID.

How to define the route

The route is the geometric backbone of any robot field test protocol. Do not describe it only in prose. Describe it in coordinates.

1. Waypoints: list WGS84 lat/lon with at least 6 decimal places. At the equator, this corresponds to a coordinate resolution of approximately 0.11 m; actual positional accuracy depends on the measurement method, and longitude resolution varies with latitude. Include start, end, and intermediate turn points.
2. Path type: straight line, closed loop, figure-eight, or waypoint mission. Closed loops allow endpoint-displacement error to be measured, provided that final position is measured against an appropriate reference.
3. Length and duration: total meters and expected traverse time at the nominal speed. State the commanded speed and the platform configuration rather than relying on a generic platform specification.
4. Surface classification: mowed grass, gravel, packed dirt, asphalt, or mixed. Note slope in degrees from a phone inclinometer if you lack a survey tool.
5. Obstacles: static (curbs, posts) marked with coordinates; dynamic obstacles excluded unless they are the variable under test.

Save the route as a GeoJSON or KML file and version it with the protocol document. That file is what a reviewer cites, not a screenshot.

What initial conditions to record

Initial conditions are the variables you fix or measure before pressing “start.” Skipping this step is a common reason field results do not replicate. Record the following before every run, not just once per session:

• Battery voltage at rest and battery state of charge where available. Declare thresholds appropriate to the battery chemistry, pack configuration, and manufacturer limits; reject runs starting below that threshold.
• Tire pressure or wheel condition if pneumatic; wear photo if solid.
• Payload mass and center of mass in grams and approximate position.
• Ambient temperature, wind speed, and precipitation from a nearby weather station or handheld anemometer.
• GNSS conditions: number of satellites, HDOP, and RTK fix status at the start point.
• Software versions: ROS 2 distribution (e.g. Humble), commit hash of your stack, firmware version of the rover core.
• Time and date in UTC, so solar position and shadow conditions are recoverable.

How many repetitions you need

One run is an anecdote. Three runs may let you spot a gross error. For a defensible robot field test protocol, 10 valid runs per condition is a practical starting point, but it is not a universal statistical minimum. The required number depends on the expected effect size, metric variance, desired statistical power, and analysis plan; determine it with a power analysis when you are testing a specific hypothesis.

If you compare two configurations (say, two path planners), run the planned number of valid trials of each, interleaved or randomized to spread out battery and weather drift. Interleaving matters: running all of condition A in the morning and condition B in the afternoon can confound the planner with the sun angle.

What to measure and how to log it

Decide the measured quantities before the first run. Adding metrics later invites cherry-picking. A minimal set for outdoor UGV navigation includes:

• Cross-track error: lateral deviation from the planned path, computed from time-synchronized position data at a declared sampling rate, reported as RMS and max.
• Along-track completion: percentage of planned distance covered before stop or timeout.
• Return-to-origin error (loops only): Euclidean distance from start to final pose, measured against the declared reference.
• Traverse time: from first motion to reaching the goal within a defined radius (e.g. 0.5 m).
• Energy consumed: integral of measured battery current over the run, in ampere-hours, using a calibrated current measurement where possible.
• Intervention count: number of times a human took manual control or restarted a node.

Log everything through ros2 bag record -a or a filtered topic list. Store the bag with the run ID, the SHA-256 checksum, and the protocol version. If you use Leo Rover with its ROS 2 software stack, verify the available topic names and message contents in your specific configuration; commonly useful data include commanded velocity, odometry, IMU, GNSS, and battery telemetry.

Ground truth without a survey team

Cross-track error requires a truth reference. For small outdoor rovers, an RTK GNSS receiver can provide centimetre-level horizontal accuracy in favourable open-sky conditions with suitable corrections and a validated setup. If RTK is unavailable, a total station or surveyed ground markers can provide a reference path; photographs can support deviation estimates only when the camera geometry and marker positions are adequately calibrated. Document the method and do not switch it mid-experiment.

When to reject a run

Rejection criteria must be declared before the runs, not after. Otherwise, you are silently filtering data. Standard rejection triggers for an outdoor field test protocol include:

• Loss of the position reference required for the primary metric for more than a declared duration, such as 5 consecutive seconds.
• Battery voltage or state of charge drops below the declared safe operating limit mid-run.
• Wind gust above your declared limit (e.g. 8 m/s for lightweight platforms).
• Rain starts during a “dry” condition trial.
• Human intervention that was not part of the tested policy.
• Any node crash or E-stop event, if it prevents completion of the planned autonomous run.
• A pedestrian, animal, or vehicle enters the test area and materially affects safety or the planned route.

Rejected runs are still logged, with the reason. Report the ratio of valid to attempted runs. A protocol that rejects 60% of attempts is telling you something about the platform, the environment, or the suitability of the test bounds.

What the result sheet looks like

A structured result sheet turns a folder of ROS 2 bags into a table you can put in a paper. A useful template contains one row per run and the following columns:

• Run ID, date, UTC time, protocol version
• Route file hash, condition label
• Initial battery voltage, temperature, wind, satellites, HDOP
• Software commit hash, firmware version
• Cross-track RMS, cross-track max, completion %, return error, traverse time, energy Ah, interventions
• Valid/rejected flag, rejection reason
• Bag file path, SHA-256

A blank CSV template with these columns is small enough to include as supplementary material with any publication. Keep it under version control alongside the protocol document.

How to report results so they can be cited

Reporting is where many field studies lose credibility. Present the mean and standard deviation for each metric where those summaries are appropriate, the number of valid runs, and the rejection ratio. For small samples or skewed data, also consider reporting individual values, medians, ranges, or confidence intervals. Include a plot of individual run trajectories over the reference path, not just an aggregate curve. Publish the raw bags or, if size prohibits, the derived per-sample CSVs and sufficient metadata to reproduce the analysis.

State the protocol version explicitly: “Test executed per Protocol v1.2, 12 attempted runs, 10 valid, 2 rejected (GNSS loss).” That sentence helps a reader identify the procedure used. It also tells you next year whether the numbers moved because the robot changed or the protocol changed.

How this scales from a Leo Rover to a fleet

The same protocol works whether you run one rover or ten. For a fleet, add a rover ID column and randomize which unit runs which condition. That controls for per-unit hardware variance (motor calibration, IMU bias, tire wear). Small platforms such as Leo Rover can make multi-unit statistical work more practical in research and education settings than a full-size UGV.

FAQ

How many runs are enough for publication?

There is no fixed number that is enough for every publication. Ten valid runs per condition is a practical starting point for many outdoor navigation studies, but the appropriate count depends on variance, expected effect size, and the planned analysis. Use a power analysis where possible, and report all attempted and valid runs.

Do you need RTK GNSS for a field test protocol?

Not always. RTK can provide centimetre-level ground truth in favourable conditions, which is useful for cross-track error at the centimetre level. For meter-scale tasks, standard GNSS plus surveyed or otherwise validated visual markers on the ground may be sufficient. Declare your ground truth method in the protocol.

What if weather changes mid-session?

Stop the session, log the reason, and resume when conditions return within declared bounds. Runs completed under out-of-bound weather are rejected per your criteria, not silently discarded.

Should you interleave conditions or block them?

Interleave or randomize condition order. Blocking (all A, then all B) can confound the tested variable with time-varying factors: battery state, sun angle, and ground moisture. Record the order used.

How do you handle software updates between runs?

Freeze the software stack for a test campaign. Pin ROS 2 package versions and record the commit hash of your own code. If an update is unavoidable, treat post-update runs as a new condition.

What if the robot fails to complete the route?

Log the run as valid with completion percentage less than 100%, unless failure was caused by a declared rejection trigger. A robot that reaches 60% of the goal is data, not a failed experiment.

Can this protocol apply indoors?

Yes, with adjustments. Replace GNSS conditions with lighting and motion-capture or other localization status. The rest of the structure – route file, initial conditions, N runs, rejection criteria, logged metrics – transfers directly.

Put the protocol to work

A written robot field test protocol is one of the cheapest instruments in your lab. It costs an afternoon to draft and saves weeks of arguing about which run “counts.” Adapt the structure above, version it, and cite it in every paper and report. If you are building a research program around small outdoor UGVs and want to see how Leo Rover is used in field studies with ROS 2, explore Fictionlab’s research applications page for platform details and reference projects.