Remote Robot Monitoring: Telemetry, Health Checks and Alerts for Robots in the Field

TL;DR: Remote robot monitoring telemetry is the practice of continuously collecting, transmitting, and evaluating a robot’s internal state so you can detect failures before they strand the platform. For a field-deployed UGV, the minimum useful set covers battery voltage and current, component temperatures, CPU and memory load, link quality, and the liveness of critical ROS 2 nodes. ROS 2 ships with a diagnostics stack (diagnostic_updater, diagnostic_aggregator, and the diagnostic_msgs/DiagnosticArray message) that standardizes how components report health. Over weak links, you downsample, aggregate, and switch transports – MQTT or a store-and-forward buffer instead of raw DDS over a VPN. Alert thresholds should be tiered (warn / error / stale) and tied to an action, not just a notification.

Why does remote robot monitoring telemetry matter for field robots?

A robot on a bench is easy to debug. You plug in a monitor, run ros2 topic echo, and watch the logs scroll. A robot 400 meters away in a quarry, inside a sewer, or on a research plot is a different problem entirely. When something goes wrong, you usually learn about it twice: once when the robot stops responding, and again when you walk out to retrieve it.

Remote robot monitoring telemetry closes that gap. Instead of binary “online / offline” status, you get a continuous stream of internal state that lets you distinguish between a depleted battery, a thermally throttled compute module, a crashed localization node, and a Wi-Fi dropout. Those four failures look identical from the operator console if all you monitor is connectivity.

The practical payoff is specific. You cut recovery trips. You catch degradation trends – a motor driver running hotter each week, a battery pack whose voltage sags earlier under the same load – before they become field failures. And you get the data needed to reproduce a fault, which matters more than most teams expect, because intermittent field faults rarely reproduce in the lab.

This applies whether you are running a single Leo Rover for a university research project or a fleet of customized platforms on an industrial site. The architecture scales down more easily than it scales up, so it is worth designing the monitoring layer early.

What should you actually monitor on a mobile robot?

Monitoring everything is a trap. Every signal you stream costs bandwidth, storage, and – most importantly – operator attention. A dashboard with 200 metrics is a dashboard nobody reads. The useful approach is to define a small core set that covers the failure modes that actually strand robots, then add domain-specific signals on top.

The following categories cover the majority of recoverable field failures on a wheeled UGV:

• Power and battery. Pack voltage, current draw, and estimated state of charge. Voltage under load is more informative than open-circuit voltage, because sag reveals cell degradation and high-current stalls. If your battery management system exposes per-cell data, log it – but you rarely need to stream it continuously.
• Temperatures. Compute module (SoC) temperature, motor driver temperature, and ambient enclosure temperature if you have a sensor for it. Thermal throttling is a silent performance killer: the robot keeps running, but perception and control loops start missing deadlines.
• Compute load. CPU utilization per core, memory usage, swap activity, and disk space remaining. Disk full is an embarrassingly common cause of field failure when rosbag recording is enabled and nobody set a rotation policy.
• Connectivity. Link quality (RSSI or SNR), round-trip latency to the base station, and packet loss. Track these even when the link is good, because the trend tells you where coverage degrades along a route.
• Node and topic liveness. Which ROS 2 nodes are alive, and whether critical topics are publishing at their expected rate. A node that crashed and was restarted by a supervisor looks healthy in a process list but may have lost its state.
• Actuator and drivetrain health. Wheel encoder consistency (do the four wheels report plausible velocities relative to each other?), commanded versus measured velocity error, and current spikes that indicate a stall or obstruction.
• Sensor validity. Frame rate on cameras, scan rate on lidar, GNSS fix type and satellite count, IMU saturation flags. A sensor that publishes garbage at the right rate is harder to catch than one that stops publishing.

For inspection-oriented deployments, where the robot’s job is to gather data rather than move payloads, sensor validity usually sits at the top of the priority list. There is no value in a successful patrol that produced unusable imagery. If that matches your use case, the inspection applications overview covers the operational side of those missions.

How often should each signal be sampled?

Different signals have different useful rates, and matching rate to purpose is the single biggest lever on bandwidth. Temperatures change on a timescale of tens of seconds, so 0.2-1 Hz is plenty. Battery voltage benefits from slightly faster sampling – around 1 Hz – because you want to catch sag during high-current maneuvers. Node liveness checks at 1 Hz are common in ROS 2 diagnostics. Link quality at 1 Hz. Wheel current, if you are hunting stalls, may need 10 Hz or more, but you should aggregate it locally and only transmit statistics unless a threshold is crossed.

How does ROS 2 diagnostics structure health reporting?

ROS 2 does not leave remote robot monitoring telemetry entirely to you. The diagnostics stack provides a defined contract between components that produce health information and tools that consume it, which means you can mix first-party drivers, third-party packages, and your own nodes without inventing a message format.

The core message type is diagnostic_msgs/DiagnosticArray. It carries a timestamp and a list of DiagnosticStatus entries. Each status has four fields that matter: a level (OK, WARN, ERROR, or STALE), a name identifying the component, a human-readable message, and a list of key-value pairs for arbitrary structured data. That key-value list is where you put the actual numbers – temperature in Celsius, publish rate in Hz, voltage in volts.

Three components do most of the work:

1. diagnostic_updater is a library you link into your own nodes. You register task functions that fill in a DiagnosticStatusWrapper, and the updater calls them on a timer and publishes to /diagnostics. It also provides ready-made helpers such as FrequencyStatus and TimeStampStatus, which check that a topic publishes within an expected rate band and that message timestamps are within acceptable limits.
2. diagnostic_aggregator subscribes to /diagnostics, groups raw statuses into a hierarchy defined by a YAML analyzer configuration, and publishes a rolled-up summary on /diagnostics_agg. This is what turns forty individual statuses into “Sensors: ERROR, Power: OK, Compute: WARN.”
3. rqt_robot_monitor renders that aggregated tree as a GUI, which is useful during development and bring-up even if your production monitoring runs elsewhere.

The hierarchy from the aggregator is the part most teams underuse. Defining analyzer groups – power, drivetrain, perception, compute, comms – gives you a top-level status for each subsystem. Your alerting layer can then operate on five summary values instead of parsing every raw status, and an operator can drill down only when something goes amber.

What about node liveness specifically?

ROS 2 does not automatically tell you when a node dies. A node that exits cleanly simply stops appearing in the graph; one that hangs may still appear but publish nothing. Two complementary techniques cover this.

First, use FrequencyStatus from diagnostic_updater on every critical topic. If the expected rate is 30 Hz and you receive 2 Hz, the status goes WARN or ERROR with the measured rate attached. This catches hangs, not just crashes.

Second, use the STALE level. When diagnostic messages for a status stop arriving, the aggregator can mark its entry STALE after a configurable timeout. STALE is semantically different from ERROR: ERROR means “this component reports a problem,” STALE means “this component stopped reporting.” Treat them differently in your alerting, because STALE often indicates a process or transport failure rather than a hardware fault.

For nodes built as ROS 2 managed (lifecycle) nodes, you also get explicit state transitions – unconfigured, inactive, active, finalized – which you can surface in telemetry. That is useful for staged bring-up, where you want to know whether navigation failed to activate or activated and then failed.

How do you transmit telemetry over a weak link?

The naive approach is to run DDS over a VPN and subscribe to /diagnostics from the operator laptop. This works on a bench and on a strong Wi-Fi link. It can degrade on LTE, long-range Wi-Fi at the edge of coverage, or any link with high jitter, because DDS discovery traffic and reliable QoS retransmissions may add load exactly when the link is already struggling.

A more robust architecture separates the on-robot domain from the uplink. Here is the pattern in text form, since it is easier to reason about than a picture:

On-robot layer: ROS 2 nodes publish raw diagnostics on /diagnostics at native rates. The diagnostic_aggregator rolls them up to /diagnostics_agg. A local recorder writes full-rate data to disk with rotation.

Edge agent layer: A single bridge node subscribes to the aggregated diagnostics plus a small set of hand-picked raw topics. It downsamples, serializes to a compact format (JSON, CBOR, or Protobuf), and buffers to local storage. Nothing else crosses the boundary.

Uplink layer: The agent publishes to a broker over MQTT (or an equivalent lightweight protocol), using a persistent session and local buffering to survive brief disconnections. MQTT QoS 1 provides at-least-once delivery between the client and broker. On reconnect, the buffer drains. Full-rate data stays on the robot and is pulled on demand or offloaded when the robot returns to a high-bandwidth zone.

Backend layer: The broker feeds a time-series database and an alerting engine. Dashboards read from the database. Operators see state; they do not maintain a live DDS session with the robot.

Several techniques reduce payload size without losing diagnostic value:

• Send on change, with a heartbeat. Transmit a signal only when it moves outside a deadband, plus a periodic heartbeat so you can distinguish “unchanged” from “disconnected.”
• Aggregate before transmitting. Instead of 10 Hz motor current, send min, max, mean, and standard deviation over a one-second window. That is four floats instead of ten, and it preserves spike information.
• Tier the data. Define a small critical set that always goes out (battery, top-level subsystem status, position) and a larger diagnostic set that is sent only when a fault triggers or when the link is good.
• Compress and batch. Batching several seconds of telemetry into one compressed message reduces per-message overhead, at the cost of latency. For health monitoring, a few seconds of latency is acceptable; for teleoperation it is not, so keep those streams separate.
• Use store-and-forward for anything you cannot lose. Fault events, mission completions, and error logs should be buffered to disk and acknowledged end-to-end, not fire-and-forget.

One design decision worth making explicitly: teleoperation and telemetry have different requirements. Teleop needs low latency and tolerates loss. Telemetry tolerates latency and should not tolerate loss. Running them over the same transport with the same QoS settings forces a compromise that serves neither well.

What thresholds should trigger an alert?

Threshold design is where most remote robot monitoring telemetry systems fail in practice. Thresholds that are too tight produce alert fatigue; operators mute the channel and miss the real fault. Thresholds that are too loose fire only after the robot has already stopped.

A workable approach uses three principles. Set thresholds from measured baselines on your own hardware in your own environment, not from datasheet maxima. Use two levels per signal, so a warning gives you time to act before an error forces a stop. And require a signal to stay out of range for a defined duration before alerting, which filters transient spikes.

Avoid copying numeric thresholds from blog posts, including this one. Instead, derive them. A reasonable procedure:

1. Baseline. Run representative missions and record the full distribution of each signal. Note the typical range and the observed extremes during normal operation.
2. Set the warning level somewhere outside the normal operating distribution but well inside the hardware limit – far enough that normal variation does not trip it, close enough that you get useful lead time.
3. Set the error level relative to the manufacturer-specified limit for that component, with margin. For temperatures, consult the official datasheet for the SoC, motor driver, or battery chemistry you are actually using; do not assume.
4. Add hysteresis and dwell time. Require the condition to hold for several consecutive samples before alerting, and require it to clear by a margin before the alert resets. Without hysteresis, a signal sitting on the boundary produces a flood of alerts.
5. Attach an action to every alert. If you cannot state what an operator should do when an alert fires, the alert should probably be a logged metric instead.

The action-mapping step deserves emphasis. Useful alert responses on a UGV typically fall into a few categories: notify only, reduce speed or duty cycle, abort the current task and return to base, or stop immediately and hold. Mapping each error-level condition to one of these turns your monitoring system from a dashboard into a safety and availability mechanism. Battery below a return-to-base reserve triggers return. SoC temperature at throttle threshold triggers duty-cycle reduction. Loss of localization confidence triggers stop and hold.

How do you handle alerts when the robot is offline?

If the uplink drops, no alert reaches the operator – which means critical safety logic cannot live in the backend. The on-robot agent must evaluate error-level conditions locally and execute the mapped action regardless of connectivity. The backend handles notification, trending, and fleet-level analysis. Treat the cloud as an observer, not a controller, for anything that must work when the link is down.

On reconnection, the buffered telemetry drains and the backend reconstructs what happened during the blackout. This is where store-and-forward pays for itself: without it, your most interesting failures are also your least documented.

How does this apply to a specific platform?

The principles above are platform-agnostic, but implementation details depend on what the hardware exposes. A platform that publishes battery voltage, wheel odometry, and motor controller state over a documented ROS 2 interface gives you most of the core signal set without additional hardware. A platform that treats its firmware as a black box forces you to add external sensing.

This is one of the practical arguments for open-source robot platforms in research and integration work: you can see which signals exist, extend the firmware to expose more, and add diagnostic_updater tasks to the driver nodes rather than wrapping them from outside. For consistency with the rest of the ROS 2 ecosystem, publishing to /diagnostics rather than a custom topic means off-the-shelf tools work immediately.

When a monitoring requirement exceeds what a standard platform exposes – additional thermal sensing, a redundant uplink, a dedicated watchdog MCU that survives SoC crashes, or integration with an existing site SCADA system – it becomes a hardware and firmware question rather than a software one. Fictionlab’s team handles that class of work as part of custom robotics projects, where the telemetry architecture is specified alongside the mechanical and electrical design rather than retrofitted afterward.

Frequently asked questions

What is the difference between telemetry and diagnostics in ROS 2?

Telemetry is the broader category: any measured data transmitted from the robot, including sensor readings, pose, and mission state. Diagnostics is a specific subset concerned with component health, standardized in ROS 2 through diagnostic_msgs/DiagnosticArray and the diagnostic_updater / diagnostic_aggregator packages. In practice, your remote robot monitoring telemetry stream usually carries aggregated diagnostics plus a handful of operational signals like position and battery state.

Can I just run ROS 2 over a VPN and subscribe to topics remotely?

You can, and for a strong, low-latency link it is the fastest path to a working setup. It can degrade on constrained links because DDS discovery traffic and reliable-QoS retransmission may add load under packet loss and jitter. For field deployments on LTE or marginal Wi-Fi, a bridge node that converts selected topics to MQTT or a similar lightweight protocol gives you more controllable behavior, including buffering during disconnections.

How much bandwidth does basic health telemetry need?

It depends entirely on what you include and at what rate, so treat any single number as unreliable. The useful framing is this: a core health set of roughly 20-40 scalar values at 1 Hz, serialized compactly, is a very small payload compared with any video or point-cloud stream. The bandwidth problem in field robotics is often sensor data, not health telemetry. Measure your own payload sizes rather than assuming.

Should alert logic run on the robot or in the cloud?

Both, with a clear division. Safety-critical and availability-critical logic – low battery reserve, thermal limits, loss of localization – must run on the robot so it works when the uplink is down. Trending, fleet comparison, notification routing, and anything requiring historical context runs in the backend. Duplicating simple threshold checks in both places is acceptable; relying solely on the cloud for stop conditions is not.

How do I detect a ROS 2 node that is running but stuck?

Process-level checks will not catch this, because the process is alive. Use FrequencyStatus from diagnostic_updater on the topics that node publishes, with an expected rate band. If the measured rate falls below the lower bound, the diagnostic status goes WARN or ERROR with the actual rate included. Combine this with TimeStampStatus to catch cases where messages publish at the right rate but carry timestamps that are too old or otherwise outside the expected range.

What does the STALE diagnostic level actually mean?

STALE indicates that diagnostic messages for a status have stopped arriving within the aggregator’s configured timeout. It is distinct from ERROR, which means the component actively reports a problem. STALE typically points to a crashed node, a dead process, or a transport failure rather than a hardware fault, so it usually warrants a different response – a restart attempt or a connectivity check rather than a hardware inspection.

Do I need a separate watchdog microcontroller?

Not for every deployment. It becomes worth considering when a crash or hang of the main compute module would leave the robot in an unsafe or unrecoverable state, since software running on that module cannot report its own failure. An independent MCU that monitors a heartbeat from the main computer and can cut motor power or trigger a reset covers that gap. For a research platform operating within line of sight, it may be unnecessary if other risk controls are adequate; for autonomous operation in an industrial setting, it is often appropriate.

Where to go from here

If your monitoring requirements go beyond what a standard platform exposes – redundant uplinks, additional sensing, a hardware watchdog, or integration with existing site infrastructure – those decisions are easier to make at the design stage than after the first field deployment. Fictionlab’s team scopes telemetry and health-monitoring architecture as part of custom robotics development. Share your operating environment, link conditions, and the failure modes you need to catch, and the conversation can start from concrete requirements rather than a generic feature list.