AI Framework Extracts Hidden Robot Actions From Ordinary Online Videos
Technology

AI Framework Extracts Hidden Robot Actions From Ordinary Online Videos

By repurposing pretrained video generation models into dual inverse and forward dynamics engines, researchers enable zero-shot action transfer across robotic embodiments.

By Asif Iqbal
Published:
Email this Article
AI Framework Extracts Hidden Robot Actions From Ordinary Online Videos
Zero-shot cross-context camera navigation. O2LaF better reproduces reference camera and agent action while preserving the target scene. Yuxin Jiang et al. / arXiv

Videos show how the world changes over time, but they rarely reveal the hidden physical commands that cause those shifts. Researchers have now developed a new artificial intelligence framework that extracts these missing action controls directly from raw, unannotated video clips by repurposing modern generative video models.

The system, called O2LaF, builds on the principle of using a single pretrained video diffusion model to handle two separate tasks at once, namely inferring what action happened and predicting what visual consequence it will produce. The authors describe their findings in a study published on the arXiv preprint server, outlining how pretrained video architectures can be adapted to discover transferable movement dynamics without requiring explicit human supervision.

How One Model Handles Two Different Dynamics

Traditional approaches to latent action learning typically rely on building separate models from scratch to map visual transitions. The authors noted that modern video diffusion transformers are already pretrained at massive scales to understand how visual scenes evolve across time. By taking a 5-billion-parameter video model, the team initialized two distinct components from the exact same pretrained weights.

Figure 2: Paradigms for self-supervised latent-action learning. Existing methods learn latent actions (a) from scratch [8, 85, 20], (b) on frozen visual features [9, 21], (c) with a pretrained video generator as the FDM only [62, 76], or (d) with an external video prior as auxiliary supervision [35]. Ours repurposes one pretrained video DiT as both the IDM and the FDM.
Paradigms for self-supervised latent-action learning. Existing methods learn latent actions (a) from scratch [8, 85, 20], (b) on frozen visual features [9, 21], (c) with a pretrained video generator as the FDM only [62, 76], or (d) with an external video prior as auxiliary supervision [35]. Ours repurposes one pretrained video DiT as both the IDM and the FDM. Credit: Yuxin Jiang et al. / arXiv

The first component acts as an inverse dynamics model, functioning as a single-pass encoder that reads video transitions and outputs a compact 32-dimensional continuous latent action vector. The second component serves as a forward dynamics model, taking those inferred actions alongside an initial video frame to generate and predict future visual states. To keep the two systems from taking shortcuts, the researchers introduced careful architectural constraints, including frame-causal attention mechanisms and high-noise training schedules that force the forward model to rely strictly on the action vector rather than copying visual details from nearby frames.

Testing Control Across Cameras and Humanoid Robots

The research team evaluated the learned action representations across multiple domains, including discrete camera navigation and continuous robotic manipulation tasks. When tested on camera motion benchmarks, the system achieved a high degree of control decodability using only a tiny fraction of labeled examples. Furthermore, when applied to egocentric human demonstration videos from the EgoDex dataset, the model successfully transferred its learned dynamics to a physical GR-1 humanoid robot without any target-specific adaptation.

In quantitative evaluations, the framework achieved an end-effector regression score of 0.277 on robotic manipulation tasks, outperforming previous baseline models that scored 0.149. The authors also demonstrated zero-shot cross-context action transfer, where an action sequence extracted from one video successfully guided video generation in an entirely new visual setting with different background appearances and camera viewpoints.

Figure 3: Action-aware response in a pretrained video DiT. Cross-attention follows the active hands and manipulated regions as the interaction changes.
Action-aware response in a pretrained video DiT. Cross-attention follows the active hands and manipulated regions as the interaction changes. Credit: Yuxin Jiang et al. / arXiv

Limitations and Future Directions

Despite strong performance across few-shot action probing and cross-embodiment tasks, the authors noted several limitations that require further investigation. The underlying structure and interpretability of the 32-dimensional latent actions remain difficult to parse directly, often requiring researchers to inspect the corresponding reference videos to understand what specific motion a vector represents.

Additionally, because the framework relies on a heavy video diffusion backbone, it is currently optimized for visual prediction rather than real-time decision-making and planning. While the forward model can function as a policy verifier to check simulated robot trajectories against real-world photos before execution, bridging high-resolution video generation with efficient decision-making dynamics remains an important challenge for future robotics research.

The study was posted to the arXiv preprint server on November 19, 2026.

Fact Checked

This article has been fact checked for accuracy, with information verified against reputable sources. Learn more about us and our editorial process.

Last reviewed on .

Article history

  • Latest version

Reference(s)

  1. Jiang, Yuxin. “One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning.” arXiv.org <http://arxiv.org/abs/2611.26617v1>.

Cite this page:

Iqbal, Asif. “AI Framework Extracts Hidden Robot Actions From Ordinary Online Videos.” BioScience. BioScience ISSN 2521-5760, 20 November 2026. <https://www.bioscience.com.pk/en/subject/technology/one-prior-two-dynamics-repurposing-video-diffusion-transformers-for-self-supervised-latent-action-learning>. Iqbal, A. (2026, November 20). “AI Framework Extracts Hidden Robot Actions From Ordinary Online Videos.” BioScience. ISSN 2521-5760. Retrieved November 20, 2026 from https://www.bioscience.com.pk/en/subject/technology/one-prior-two-dynamics-repurposing-video-diffusion-transformers-for-self-supervised-latent-action-learning Iqbal, Asif. “AI Framework Extracts Hidden Robot Actions From Ordinary Online Videos.” BioScience. ISSN 2521-5760. https://www.bioscience.com.pk/en/subject/technology/one-prior-two-dynamics-repurposing-video-diffusion-transformers-for-self-supervised-latent-action-learning (accessed November 20, 2026).
End of the article