VLA (Vision-Language-Action model)

VLA (Vision-Language-Action model) – definition

A Vision-Language-Action (VLA) model is a multimodal machine-learning model that converts visual observations and natural-language instructions into actions. In mobile robotics, a VLA can receive images, depth data, robot state and a task prompt, then produce a high-level navigation command, a discrete behavior selection or a low-level motion command.

For an unmanned ground vehicle (UGV), a VLA is not a replacement for the complete autonomy stack. It is normally an additional decision layer above safety-critical control, state estimation and collision avoidance. A mobile robot still requires validated odometry, a defined coordinate-frame tree, actuator limits and an independent stop mechanism. ROS 2 does not define a standard VLA interface, message type or reference implementation.

The term is commonly used in robotics research for models trained on paired visual observations, language descriptions and action trajectories. RT-2, published by Google DeepMind in 2023, is a widely cited example of a vision-language-action model. Its authors describe an approach that transfers visual-language knowledge to robotic control through action-token prediction. Source: Google DeepMind, “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control”, 2023.

VLA models in a UGV autonomy stack

On a Leo Rover or Raph Rover, a VLA should be treated as a task-level component. It can interpret an instruction such as “inspect the blue valve and stop one metre before it” and convert it into structured goals for ROS 2 nodes. Deterministic navigation and safety layers should remain responsible for feasible velocity generation and obstacle handling.

A practical separation of responsibilities is shown below. This architecture limits the authority of a probabilistic model over wheel motion.

Layer	Typical ROS 2 interface	VLA role
Perception	`sensor_msgs/msg/Image`, `sensor_msgs/msg/CameraInfo`, `sensor_msgs/msg/PointCloud2`	Interprets semantic content, such as objects, signs or terrain features.
Localization	`nav_msgs/msg/Odometry`, `geometry_msgs/msg/PoseWithCovarianceStamped`	Consumes robot pose and uncertainty as context. It should not estimate pose without a validated localization pipeline.
Task planning	`nav2_msgs/action/NavigateToPose`	Generates a target pose, waypoint sequence or named behavior.
Motion control	`geometry_msgs/msg/Twist` on `/cmd_vel`	Should normally be mediated by Nav2, a velocity limiter or a safety supervisor.

REP 105 defines the standard ROS coordinate-frame conventions used by mobile robots. In particular, the map, odom and base_link frames separate a world-fixed reference frame, locally continuous odometry and the robot body frame. A VLA node that returns spatial goals must use explicit frame identifiers and timestamps rather than ambiguous phrases such as “move forward”. Source: ROS Enhancement Proposal 105, “Coordinate Frames for Mobile Platforms”.

Inputs, outputs and action representations

VLA inputs are usually synchronized observations from one or more sensors plus a textual instruction. Camera images can provide semantic context, while LiDAR, IMU and wheel odometry provide geometry and motion information that visual models alone may not recover reliably.

For UGV integration, outputs should be constrained to a small and inspectable action space. Common output forms include the following.

Navigation goal: a geometry_msgs/msg/PoseStamped submitted to Nav2.
Semantic waypoint: a selected location from a pre-defined map database, for example loading_zone or inspection_point_03.
Behavior command: a controlled transition such as start_inspection, pause or return_home.
Velocity proposal: linear and angular velocity values that must be checked against platform limits before publication to /cmd_vel.

A direct velocity representation can be written as a_t = [v_x, ω_z], where v_x is forward velocity and ω_z is yaw rate. For a differential-drive UGV, these values must be converted by the base controller into left and right wheel commands. The conversion is platform-specific and should remain outside the VLA model.

ROS 2 integration pattern

A ROS 2 VLA integration commonly uses a dedicated node that subscribes to observations, calls a local or remote inference service, validates the returned structured action and forwards an accepted goal to Nav2. JSON schema validation or a fixed ROS action schema is preferable to free-form text parsing.

The example below launches a hypothetical VLA bridge. It accepts camera data and robot state, but sends only validated goals to the navigation stack.

vla_bridge:
  ros__parameters:
    image_topic: /camera/color/image_raw
    camera_info_topic: /camera/color/camera_info
    odom_topic: /odom
    instruction_topic: /mission_instruction
    output_mode: navigate_to_pose
    target_frame: map
    require_tf_transform: true
    use_nav2_action: true
    reject_unknown_frames: true

QoS selection matters for sensor streams. ROS 2 provides predefined QoS profiles, including sensor-data-oriented settings intended for timely delivery where occasional loss is preferable to delayed data. Nodes must use compatible QoS policies or a subscription may receive no messages. Source: ROS 2 documentation, “About Quality of Service settings”.

Use with Leo Rover and Raph Rover

Leo Rover is a four-wheel, skid-steered mobile platform with IP55 protection and an integrated Raspberry Pi compute unit. It can run ROS 2, including Humble, on its integrated computer. A VLA workload may exceed the resources available on the default compute unit, particularly when image encoding, object detection, local inference and Nav2 run concurrently. In such cases, inference can be moved to an external computer while control and safety nodes remain on the rover. Source: Leo Rover technical documentation, docs.fictionlab.pl.

Leo Rover is not autonomous out of the box. A deployment requires integration of localization, navigation, sensor drivers and safety handling before a VLA can issue meaningful tasks. Typical research use cases include language-directed indoor inspection, semantic waypoint selection and dataset collection for perception experiments.

Raph Rover can accommodate applications requiring greater payload capacity, such as a larger compute enclosure, multi-camera rig, LiDAR or RTK GNSS receiver. The VLA integration principles are unchanged: sensor calibration, time synchronization, localization quality and independent command limits remain necessary. Raph Rover is not a substitute for Leo Rover in laboratory and educational deployments where a smaller platform is appropriate.

Limitations and verification

VLA models can produce plausible but incorrect outputs. They can also fail when lighting, terrain, object appearance or language differs from the training data. A VLA should therefore not be the sole source of obstacle detection, localization or emergency stopping.

Before field operation, engineers should define measurable acceptance criteria and test the system in a controlled environment. The verification plan should include the following checks.

Validate that every generated pose contains a valid ROS frame ID and timestamp.
Reject goals outside a configured geofence or occupancy-map boundary.
Measure task success separately from navigation success and collision-avoidance performance.
Record ROS 2 bags containing sensor topics, transforms, VLA requests and accepted actions for reproducible analysis.
Provide a deterministic stop path that does not depend on model inference or network availability.

VLA (Vision-Language-Action model)