AI Coding Agents Struggle to Migrate Machine Learning Repositories Across Hardware Stacks

Artificial intelligence coding agents are attempting to translate entire machine learning codebases from PyTorch on NVIDIA graphics cards to JAX on Google tensor processing units, revealing a major hurdle in hardware portability. While researchers frequently want to run successful experiments on whatever hardware they have available, shifting a whole repository across software stacks involves rebuilding models, optimizers, and data pipelines under an entirely different compiler and memory architecture. To measure how well automated systems handle this challenge, computer scientists introduced a benchmark called XStack, which tests AI agents on migrating 20 real machine learning repositories.

How the Cross-Stack Benchmark Evaluates AI Coding Agents

The evaluation framework supplies each coding agent with a working PyTorch repository pinned to a specific Git commit and a clean machine powered by tensor processing units, or TPUs, which are specialized accelerator chips built by Google. The agents must install all necessary packages, write setup scripts, translate the model using libraries like Flax and Optax, and deliver a fully functional JAX repository capable of running the original experiments from scratch. To ensure fair grading without giving away the exact numerical answers beforehand, the researchers implemented a two-phase system. If an agent fails its first attempt because it ran a different valid experiment variant, a second phase reveals the primary experiment recipe while keeping the target metric hidden. Every submission is then transferred to a freshly wiped machine and executed to check whether it reaches the published benchmark results.

Why Verification Remains the Biggest Bottleneck for Code Migration

When evaluated across seven frontier models from five major artificial intelligence laboratories, the overall success rate reached just 31 percent across 140 tested cells, according to a study published in an arXiv preprint server. Claude Opus 5 led the field by successfully migrating 15 out of 20 repositories, achieving a 75 percent pass rate. GPT-5.6-sol followed with 45 percent, while DeepSeek V4.1 Flash also managed 40 percent. Other models scored significantly lower, struggling with runtime crashes, fidelity gaps where models fell short of target metrics, and verification failures. The authors noted that translation alone is rarely the hardest part, because automated agents can easily convert code when tested on identical hardware. The true difficulty lies in coordinating framework conversions alongside hardware migrations, where agents must set up unfamiliar virtual machine environments and verify that their code actually works under real-world conditions. Furthermore, the data showed that an agent’s accuracy scales predictably with its computing cost, rising by roughly 25 percentage points for every tenfold increase in expenditure.

The study was posted to the arXiv preprint server on November 5, 2026.