Glossary

Fictionlab > Glossary > Reinforcement learning

Reinforcement learning

Reinforcement learning – definition

Reinforcement learning (RL) is a machine learning method in which an agent learns a control policy through repeated interaction with an environment. At each time step, the agent observes a state, selects an action, receives a scalar reward, and transitions to a new state. The objective is to maximise expected cumulative reward rather than reproduce labelled examples.

In mobile robotics, the agent is typically a navigation or motion-control component. Its actions can be linear and angular velocity commands, waypoint-selection decisions, or local obstacle-avoidance commands. The environment includes the UGV chassis, terrain, sensors, obstacles, localisation uncertainty, actuator limits, and safety constraints.

The common formal model is a Markov decision process (MDP), defined by states S, actions A, transition probabilities P, a reward function R, and discount factor γ. The return from time step t is:

G_t = r_(t+1) + γr_(t+2) + γ²r_(t+3) + ...

A learned policy π(a|s) maps an observed state to an action distribution. For a differential-drive UGV such as Leo Rover, an action may be represented as geometry_msgs/msg/Twist, with linear.x and angular.z passed through a safety and velocity-limiting layer before reaching the base controller. RL does not make Leo Rover autonomous by default. It requires a defined observation pipeline, training process, validation procedure, and integration with the ROS 2 navigation stack.

Reinforcement learning for UGV navigation

RL is most relevant when manually designed control rules are difficult to tune. Examples include traversing variable terrain, selecting local paths in cluttered spaces, or adapting speed to perception confidence. It is usually applied to a bounded subproblem rather than replacing the complete robotics stack.

A practical UGV implementation commonly separates deterministic functions from learned functions:

  • State estimation: wheel odometry, IMU data, LiDAR scans, depth images, GNSS, or RTK-GNSS provide observations.
  • Policy inference: a trained model produces an action or a high-level navigation decision.
  • Safety supervision: collision checking, velocity limits, emergency stop logic, and command timeouts remain deterministic.
  • Control interface: the resulting command is published to the robot base through ROS 2 topics and controllers.

For mobile robots, a raw sensor observation is rarely sufficient. A policy may use a local occupancy grid, a cropped 2D LiDAR scan, robot velocity, relative goal position, and estimated clearance. This reduces dependence on visual appearance and makes simulation-to-real transfer more manageable.

How reinforcement learning fits into ROS 2

ROS 2 does not define a reinforcement learning framework. It provides communication, parameter, launch, logging, and lifecycle mechanisms that can connect an RL inference node to navigation, sensing, and simulation nodes. ROS 2 uses DDS as its default middleware abstraction and defines quality-of-service policies for publishers and subscribers. Source: ROS 2 Humble QoS documentation.

Depending on the installed software version, Leo Rover can be used with ROS 2 and uses a Raspberry Pi as its default compute unit. The supported ROS 2 distribution should be verified in the current Leo Rover documentation. Inference workload must be matched to available CPU, memory, and optional accelerator resources. Training is generally performed on a separate workstation or compute server. The deployed rover normally performs policy inference only.

rl_navigation_node:
  ros__parameters:
    model_path: "/opt/models/local_policy.onnx"
    cmd_vel_topic: "/cmd_vel"
    scan_topic: "/scan"
    odom_topic: "/odom"
    control_rate_hz: 10.0
    max_linear_velocity: 0.25
    max_angular_velocity: 0.8
    command_timeout_s: 0.3

The values above are integration examples, not Leo Rover hardware limits. Command topics and interfaces must be verified against the installed Leo Rover ROS 2 configuration in the Leo Rover technical documentation.

A minimal ROS 2 command path can use a policy node that publishes velocity commands. A separate safety node should retain authority to stop or constrain motion when sensor data is stale, obstacles violate a configured distance threshold, or localisation is invalid.

ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist \
"{linear: {x: 0.10}, angular: {z: 0.0}}"

Algorithms and evaluation metrics

Policy-gradient and actor-critic methods are frequently used for continuous robot actions. Examples include Proximal Policy Optimisation (PPO), Soft Actor-Critic (SAC), and Deep Deterministic Policy Gradient (DDPG). Value-based methods such as DQN are better aligned with discrete action spaces, for example selecting one action from a fixed set of steering commands.

Training reward design strongly affects behaviour. A navigation reward can combine progress toward a goal, collision penalties, excessive rotation penalties, energy-related costs, and terminal rewards. Reward components must be inspected individually because a high aggregate reward can conceal unsafe or inefficient behaviour.

Evaluation should use task-level and safety-level metrics:

  • success rate over a defined set of maps, terrain conditions, and initial poses;
  • collision rate and minimum observed obstacle clearance;
  • path length relative to a reference planner;
  • time to goal and command latency;
  • recovery rate after localisation loss, wheel slip, or blocked paths;
  • inference time and CPU, GPU, memory, and thermal utilisation on the deployment computer.

Simulation-to-real transfer on Leo Rover and Raph Rover

Training directly on a physical UGV is slow and can damage equipment. Simulation enables thousands of episodes with controlled obstacle layouts, sensor noise, terrain friction, wheel slip, and latency. Gazebo and other ROS-compatible simulators can publish the same conceptual sensor and velocity interfaces used by physical robots.

For Leo Rover, RL experiments should account for its four-wheel differential-drive configuration and IP55 enclosure rating. A trained policy should be tested first at restricted speed, with conservative obstacle margins and supervised operation. LiDAR-based local observations are often more robust than purely camera-based observations when illumination changes.

Raph Rover can be appropriate when a project requires greater payload capacity for compute hardware, LiDAR, depth cameras, GNSS/RTK equipment, or protected sensor assemblies. The larger platform does not replace Leo Rover in laboratory or educational workflows. The policy must be retrained or at least revalidated when vehicle mass, wheel geometry, payload position, braking behaviour, sensor placement, or controller parameters differ.

Limitations and engineering constraints

RL policies can fail outside their training distribution. Unseen terrain, reflective surfaces, dropped sensor frames, inaccurate odometry, and changes in payload can produce actions that were not evaluated during training. A policy is therefore not a substitute for localisation, collision monitoring, or an emergency-stop strategy.

ROS 2 distributions also matter. REP 2000 defines target platforms and support expectations for ROS distributions; package compatibility should be checked for the selected ROS 2 release rather than assumed from ROS 1 documentation. Source: REP 2000 – ROS 2 Releases and Target Platforms.

See also