How a robot controls its arm between camera images

Consider a robot arm lifting a box. Its camera estimates where the box is. Joint encoders report how far the arm has moved, and a sensor at the gripper measures the force applied to the box. The controller needs all three kinds of information to lift the box without dropping it.

Those sensors do not update together. In a robotics study on multiscale sensor fusion, Singh and colleagues discuss cameras operating at about 30 Hz, joint measurements at 100 Hz, and force-torque sensors at 500 Hz [13]. At those example rates, a camera provides a new image about every 33 milliseconds; the joint encoders update roughly every 10 milliseconds. The arm keeps moving between images. During that interval, the controller must estimate where the box has moved and whether the grasp is still secure. These rates come from the study and are not specifications for all robots.

Neuroscience researchers have studied a similar problem in biological movement. Wolpert, Miall, and Kawato reviewed evidence in 1998 that the cerebellum may use internal models to predict the effects of motor commands [1]. A forward model estimates where the body will be after a movement, before delayed sensory information arrives. Wolpert and Kawato also proposed using multiple paired forward and inverse models to handle different movement conditions [2]. In a robot, a comparable predictor could estimate the next position of the gripper and compare it with the next sensor reading.

The problem changes when the same instruction is given to another robot arm. Both machines may be asked to move the box to the same shelf, but they can have different reach, grippers, joint limits, and actuator commands. Copying the first robot’s joint movements is unlikely to place the second robot in the same position. The shared information is the task goal and what happened during earlier attempts, including any observed failure.

This paper proposes a controller with separate representations of the task and the robot’s physical body. It combines an estimate of the current scene with predictions of candidate movements. It can consult recorded failures when choosing the next action, but each arm must produce commands that meet its own mechanical limits. We specify the equations for those components and experiments that would show whether they improve control compared with simpler methods. The combined system has not yet been tested.

Research on predicting movement and remembering past actions

What existing robot-learning experiments have shown

Existing robotics experiments show which parts of the proposal already have engineering precedents. Banino and colleagues trained artificial agents whose representations developed grid-like properties and supported vector-based navigation [8]. MetaMorph conditioned a controller on the structure of modular robots and tested it on held-out morphologies [10]. The Open X-Embodiment collaboration assembled more than one million real-robot trajectories across 22 embodiments and reported benefits from training across robot datasets [11]. Those experiments support learned representations and transfer in their respective settings; they do not show that a motor command can be reused unchanged on any machine.

The RT-1 study at Google trained a robotics transformer on approximately 130,000 episodes spanning more than 700 tasks, collected with 13 robots over 17 months [14]. In one mixed-data evaluation, adding Kuka bin-picking data to the Everyday Robots dataset improved the reported bin-picking result on the Everyday Robots platform from 22% to 39%, while performance on its original classroom evaluation remained similar [14]. The figures come from that study’s evaluation, not from Ontos. They illustrate the value of shared training data and the difficulty of transfer between platforms.

Hafner and colleagues trained DreamerV3 to learn a predictive world model and improve behavior through imagined rollouts, reporting experiments across more than 150 tasks [12]. The Barkour project at Google offers a different point of comparison. It measured quadruped agility on a defined course with weave poles, an incline, a jump, and end tables; its controller combined learned locomotion policies with recovery behavior [15]. The study assessed success, speed, and failures on real hardware rather than relying on the appearance of fluent motion.

None of these studies tested the full combination proposed here. We would measure whether checking prediction errors, retrieving selected failures, and adapting a limited set of body parameters improves a predictive controller’s performance. A failure to improve on a recurrent policy or an ordinary model-predictive controller would be an important finding.

Representing the same task on two different robot arms

Moving the box to a target shelf is a task both arms can share. The motor commands needed to do it depend on the arm. We encode that difference explicitly. Let $m$ identify the robot and let $\mu_m$ describe its joints, actuator limits, sensors, and command interface. At time $t$, the physical state $s_t$ is not directly observable. The robot receives observations $o_t^m$ and issues commands $u_t^m$. We model its interaction with the environment as

$$\begin{aligned} s_{t+1} &\sim p(s_{t+1}\mid s_t,u_t^m,\mu_m),\\ o_t^m &\sim p(o_t^m\mid s_t,\mu_m). \end{aligned}$$

A single measurement rarely determines all variables needed for control. The belief state $b_t^m$ represents the distribution over possible states given the recorded observations, previous commands, and robot description:

$$b_t^m(s)=p(s_t=s\mid o_{0:t}^m,u_{0:t-1}^m,\mu_m).$$

An exact belief distribution is usually too costly to maintain for high-dimensional sensor data. We therefore propose a compact learned representation $z_t=(z_t^{\mathrm{task}},z_t^{\mathrm{body}})$ and an uncertainty estimate $\Sigma_t$. For the box task, the task portion could encode the box pose, target location, and whether the intended transfer has been completed. The body portion could encode joint positions, grasp geometry, contact estimates, and actuator response. Neither portion can be defined without regard to the other: a target pose is relevant only if the robot can reach it, and a contact estimate depends on gripper geometry.

A recurrent estimator $q_\psi$ receives the previous estimate, uncertainty, most recent command, new observation, and robot description:

$$q_\psi(z_t,\Sigma_t\mid z_{t-1},\Sigma_{t-1},u_{t-1}^m,o_t^m,\mu_m).$$

The transition and observation models predict the next state and expected sensor measurements:

$$\begin{aligned} \hat z_{t+1} &\sim p_\theta(z_{t+1}\mid z_t,u_t^m,\mu_m),\\ \hat o_{t+1}^m &\sim p_\phi(o_{t+1}^m\mid\hat z_{t+1},\mu_m). \end{aligned}$$

The equations define the information each model must receive and predict. They do not specify a trained network. An implementation must still choose state dimensions, sensor encoders, probability distributions, operating frequency, and compute limits. We expect different robot families to need different input adapters and action decoders. The claim worth testing is whether some task variables and prediction parameters can remain useful when those adapters change.

How the controller responds when a box is heavier than expected

Expand diagram ↗

The following example traces the proposed controller through a box lift. It explains the design; it is not a completed experiment. The camera estimates the box pose. The joint encoders report the arm configuration, and the gripper’s force measurement helps estimate contact. Each measurement carries its own timestamp. The estimator combines them with previous commands to maintain a current state and uncertainty.

Before sending the lifting command, the controller predicts how far the joints and box will move and the force reading it expects at the gripper. If the observed displacement is smaller than predicted, the error monitor records that mismatch. The controller then updates its state estimate, considers whether the payload model should change, and generates a new candidate action. A critical sensor loss or an action outside the allowed operating envelope goes to the safety supervisor instead of an unconstrained retry.

For measured vector $y_t$, predicted measurement $\hat y_t$, and residual covariance $S_t$, the error monitor can compute the normalized squared residual

$$r_t=(y_t-\hat y_t)^\top S_t^{-1}(y_t-\hat y_t).$$

A large residual means the sensor reading differs from the prediction more than the model’s uncertainty would suggest. The number alone cannot reveal whether the box is heavier than expected, the grasp has slipped, or a sensor is miscalibrated. Those explanations require additional measurements or a diagnostic test. Thresholds could be estimated from held-out runs; a fixed threshold chosen from a clean Gaussian model would be unreliable under the correlated errors and missing observations common on real hardware.

Training with sensor recordings and task outcomes

Each training episode would record camera images, joint and force readings, the commands sent to the robot, the robot’s mechanical description, and whether the task succeeded. Where available, object poses, contact state, or privileged simulation measurements can supervise parts of the latent state. Privileged simulation inputs must be identified and excluded from real-world evaluation inputs. The predictive model can learn by comparing its next-state forecasts with subsequent measurements; the task policy can learn from demonstrations or task rewards.

A candidate training objective combines prediction, state estimation, task performance, and cross-body alignment:

$$\mathcal L=\alpha\mathcal L_{\mathrm{prediction}} +\beta\mathcal L_{\mathrm{state}} +\gamma\mathcal L_{\mathrm{task}} +\delta\mathcal L_{\mathrm{alignment}}.$$

The expression does not identify an optimizer, a dataset split, or the exact target for each term. Those choices belong in an implementation report. They also create a genuine design question: an image-reconstruction objective may retain texture that has little effect on control, while aggressive alignment across robots may erase contact information that a particular gripper needs. We would tune the coefficients on training and validation data and then measure whether transfer survives on a robot excluded from that process.

Testing delayed camera frames, changed loads, and unfamiliar robot arms

To test transfer between robots, we would train and tune the system on two arms with different joint arrangements, then evaluate it on a third arm excluded from training. Each must be physically capable of the same object-transfer tasks. We would also evaluate locomotion on its own task suite, because success on a manipulation benchmark does not establish transfer to a legged robot. Train and test sets must be separated by robot, object identity, layout, and disturbance. Frames from one episode must remain in the same split.

We propose three pre-registered comparisons.

Delayed observations. Introduce controlled camera latency, temporary occlusion, and sensor dropout during object transport. Compare the full system with a recurrent reactive controller trained on the same episodes and compute budget. The test asks whether explicit prediction improves task completion and reduces recovery time. Report uncertainty calibration and safety interventions, since a controller can appear faster by accepting more risk.

Changed load or friction. Alter a payload or contact condition without telling the controller when the change occurs. Compare bounded local adaptation with the same system’s adaptation disabled. Measure the number of actions and elapsed time required to return to acceptable tracking, together with failed grasps and constraint violations. Both conditions should receive the same observation history and restart allowance.

Recovery on a held-out arm. Record selected failures during development, then ask the unseen arm to complete physically compatible tasks with and without access to those episodes. Count successful recoveries, repeated failed attempts, memory selections that did not change the action, and harmful retrievals. Task similarity is insufficient evidence of transfer if the retrieved action violates the new arm’s reach or gripper constraints.

A classical state estimator with model-predictive control is an additional baseline whenever the system model and measurements support one. The learned architecture should also be ablated by disabling its memory, prediction loss, or body adaptation in turn. Model size, training interactions, inference hardware, control frequency, and sensor rates must be reported for every comparison. Median and 99th-percentile inference latencies are important because occasional slow decisions can matter during contact-rich tasks.

If a recurrent policy matches the proposed system at lower compute cost, the added machinery has no demonstrated value on that benchmark. If retrieval increases repeated failures on the held-out arm, the memory interface needs revision. If adaptation improves tracking but increases safety interventions, it has failed the proposed criterion. These outcomes would be reported alongside successful trials; the experiments have not yet been run on the integrated architecture.

What the results would need to show

The neuroscience papers offer models of how predictions and stored experiences can influence movement [1, 2, 4, 5]. Robotics studies show that predictive representations and some forms of cross-robot training can work in defined experimental settings [10–15]. These are findings from different systems; the papers do not establish that combining them will produce a better controller. None has tested the full architecture specified in this paper.

The next test is specific: when camera frames are delayed, a box becomes heavier, or a new arm is used, does the complete controller finish more tasks than a recurrent policy or a classical model-predictive controller using comparable data and compute? The answer requires held-out robots, measured latency, and a record of failed actions. Until those results exist, the architecture remains a research design from Ontos Research.

References

1. Wolpert, D. M., Miall, R. C., & Kawato, M. (1998). Internal models in the cerebellum. Trends in Cognitive Sciences, 2(9), 338–347. doi:10.1016/S1364-6613(98)01221-2.
2. Wolpert, D. M., & Kawato, M. (1998). Multiple paired forward and inverse models for motor control. Neural Networks, 11(7–8), 1317–1329. doi:10.1016/S0893-6080(98)00066-5.
3. Wolpert, D. M., Goodbody, S. J., & Husain, M. (1998). Maintaining internal representations: The role of the human superior parietal lobe. Nature Neuroscience, 1, 529–533. doi:10.1038/2245.
4. Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017). The hippocampus as a predictive map. Nature Neuroscience, 20, 1643–1653. doi:10.1038/nn.4650.
5. Mattar, M. G., & Daw, N. D. (2018). Prioritized memory access explains planning and hippocampal replay. Nature Neuroscience, 21, 1609–1617. doi:10.1038/s41593-018-0232-z.
6. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599. doi:10.1126/science.275.5306.1593.
7. Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Networks, 12(7–8), 961–974. doi:10.1016/S0893-6080(99)00046-5.
8. Banino, A., Barry, C., Uria, B., et al. (2018). Vector-based navigation using grid-like representations in artificial agents. Nature, 557, 429–433. doi:10.1038/s41586-018-0102-6.
9. Rao, R. P. N., & Ballard, D. H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2, 79–87. doi:10.1038/4580.
10. Gupta, A., Fan, L., Ganguli, S., & Fei-Fei, L. (2022). MetaMorph: Learning universal controllers with transformers. International Conference on Learning Representations. Project and paper.
11. Open X-Embodiment Collaboration. (2024). Open X-Embodiment: Robotic learning datasets and RT-X models. IEEE International Conference on Robotics and Automation, 6892–6903. doi:10.1109/ICRA57147.2024.10611477.
12. Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025). Mastering diverse control tasks through world models. Nature, 640, 647–653. doi:10.1038/s41586-025-08744-2.
13. Singh, S., McCann Ramirez, F. E., Varley, J., Zeng, A., & Sindhwani, V. (2022). Continuous control and multiscale sensor fusion with Neural CDEs. IROS & RSS Imitation Learning Workshop. Google Research publication.
14. Brohan, A., et al. (2022). RT-1: Robotics Transformer for real-world control at scale. Google Research technical article.
15. Caluwaerts, K., & Iscen, A. (2023). Barkour: Benchmarking animal-level agility with quadruped robots. Google Research technical article.

Appendix: Data and code needed to reproduce the experiments

An implementation report must provide the robot description format, joint and sensor mappings, actuator command units, timestamp synchronization, calibration procedures, coordinate-frame conventions, and per-platform safety constraints. For each model, report the architecture, checkpoint, parameter count, training loss, data sources, optimizer, random seeds, and inference device. Data partitions must identify robot embodiments, objects, environments, and disturbances held out from training.

For every real-robot trial, retain time-aligned observations, controller outputs, safety-supervisor interventions, attempted and completed tasks, and reasons for terminating a run. Report the metric definitions and confidence intervals, and provide scripts that reproduce the reported aggregates from the evaluation logs. If a dataset cannot be released, describe its collection and access restrictions and provide an independently executable simulation test with matching task criteria.