AI Framework Extracts Hidden Robot Actions From Ordinary Online Videos
By repurposing pretrained video generation models into dual inverse and forward dynamics engines, researchers enable zero-shot action transfer across robotic embodiments.
Videos show how the world changes over time, but they rarely reveal the hidden physical commands that cause those shifts. Researchers have now developed a new artificial intelligence framework that extracts these missing action controls directly from raw, unannotated video clips by repurposing modern generative video models.
The system, called O2LaF, builds on the principle of using a single pretrained video diffusion model to handle two separate tasks at once, namely inferring what action happened and predicting what visual consequence it will produce. The authors describe their findings in a study published on the arXiv preprint server, outlining how pretrained video architectures can be adapted to discover transferable movement dynamics without requiring explicit human supervision.
How One Model Handles Two Different Dynamics
Traditional approaches to latent action learning typically rely on building separate models from scratch to map visual transitions. The authors noted that modern video diffusion transformers are already pretrained at massive scales to understand how visual scenes evolve across time. By taking a 5-billion-parameter video model, the team initialized two distinct components from the exact same pretrained weights.
![Figure 2: Paradigms for self-supervised latent-action learning. Existing methods learn latent actions (a) from scratch [8, 85, 20], (b) on frozen visual features [9, 21], (c) with a pretrained video generator as the FDM only [62, 76], or (d) with an external video prior as auxiliary supervision [35]. Ours repurposes one pretrained video DiT as both the IDM and the FDM.](https://www.bioscience.com.pk/images/859c3d7245.avif)
The first component acts as an inverse dynamics model, functioning as a single-pass encoder that reads video transitions and outputs a compact 32-dimensional continuous latent action vector. The second component serves as a forward dynamics model, taking those inferred actions alongside an initial video frame to generate and predict future visual states. To keep the two systems from taking shortcuts, the researchers introduced careful architectural constraints, including frame-causal attention mechanisms and high-noise training schedules that force the forward model to rely strictly on the action vector rather than copying visual details from nearby frames.
Testing Control Across Cameras and Humanoid Robots
The research team evaluated the learned action representations across multiple domains, including discrete camera navigation and continuous robotic manipulation tasks. When tested on camera motion benchmarks, the system achieved a high degree of control decodability using only a tiny fraction of labeled examples. Furthermore, when applied to egocentric human demonstration videos from the EgoDex dataset, the model successfully transferred its learned dynamics to a physical GR-1 humanoid robot without any target-specific adaptation.
In quantitative evaluations, the framework achieved an end-effector regression score of 0.277 on robotic manipulation tasks, outperforming previous baseline models that scored 0.149. The authors also demonstrated zero-shot cross-context action transfer, where an action sequence extracted from one video successfully guided video generation in an entirely new visual setting with different background appearances and camera viewpoints.

Limitations and Future Directions
Despite strong performance across few-shot action probing and cross-embodiment tasks, the authors noted several limitations that require further investigation. The underlying structure and interpretability of the 32-dimensional latent actions remain difficult to parse directly, often requiring researchers to inspect the corresponding reference videos to understand what specific motion a vector represents.
Additionally, because the framework relies on a heavy video diffusion backbone, it is currently optimized for visual prediction rather than real-time decision-making and planning. While the forward model can function as a policy verifier to check simulated robot trajectories against real-world photos before execution, bridging high-resolution video generation with efficient decision-making dynamics remains an important challenge for future robotics research.
The study was posted to the arXiv preprint server on November 19, 2026.
This article has been fact checked for accuracy, with information verified against reputable sources. Learn more about us and our editorial process.
Last reviewed on .
Article history
- Latest version
Reference(s)
- Jiang, Yuxin. “One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning.” arXiv.org <http://arxiv.org/abs/2611.26617v1>.
Cite this page:
- Posted by Asif Iqbal