New System Doubles AI Agent Speed by Reallocating Graphics Memory
TideCache optimizes hybrid language models on superchips by dynamically sharing memory pools and streaming context prefixes during idle decode cycles.
A new serving system named TideCache accelerates open-weight language models on high-performance superchips by up to two times while significantly reducing response delays. Modern hybrid models mix global-attention layers, which remember vast contexts, with bounded-state layers that maintain fixed memory footprints. Because different tasks stress these memory components unevenly, fixed allocation limits serving capacity when workloads shift. The authors propose an adaptive solution that dynamically rebalances physical graphics memory and streams context prefixes during idle hardware windows.
How Dynamic Allocation Solves Memory Bottlenecks
To overcome the limitations of rigid memory slicing, the authors developed the Transparent Cache Pool, a mechanism that manages a shared arena of physical graphics memory pages. When a specific cache component blocks incoming requests due to a shortage, the system reallocates physical pages behind fixed virtual addresses without relocating live data or altering backend interfaces. As detailed in a study published on the arXiv preprint server by Jinbin Luo and colleagues, this design preserves existing cache layouts while eliminating the idle waste common in static setups.
In evaluations across seven different model and platform configurations, the system adapted seamlessly to fluctuating demands. For instance, when running the Qwen3.8-27B model on NVIDIA GH200 superchips under a shifting workload that included a sudden traffic burst, the full setup completed 183 agent turns compared to just 115 turns for older hierarchical caching baselines. This dynamic adjustment allowed roughly half of the available graphics memory pool to shift ownership depending on current request pressures.

Overlapping Data Transfers with Active Computation
Beyond managing internal memory pools, the architecture introduces a Cache Streamer that exploits idle high-bandwidth connections between the central processing unit and the graphics processor during decoding steps. While bounded-state layers compute without needing global attention data, the system prefetches historical context prefixes from host memory directly into shared graphics memory slots. This layer-wise streaming approach hides transfer latency by overlapping it with active calculations.
To maintain high performance without interrupting execution graphs, the authors designed a graph-compatible handshake mechanism using persistent coordination flags. This setup ensures that data transfers finish just before attention layers consume them. According to the evaluation data, effective global-attention capacity for large models expanded by up to 76 percent, allowing systems to sustain larger batch sizes and higher concurrency.

Real-World Performance Gains and System Limits
When tested on advanced NVIDIA GB300 and GH200 hardware configurations, TideCache delivered substantial speedups across chat and coding agent workloads. The system completed up to 2.04 times as many agent turns, reduced overall median per-token latency by 20 to 59 percent, and improved time-to-first-token metrics by factors ranging from 1.40 to 3.05 compared to baseline serving configurations.
Despite these improvements, the approach has specific operational boundaries. The streaming capacity is fundamentally constrained by the available bidirectional interconnect bandwidth and the duration of computation intervals between layers. If transfer sizes exceed what can be hidden within a decode step, throughput gains diminish. Nevertheless, the integration of transparent memory pooling and layer-wise streaming offers a robust path forward for serving complex hybrid models efficiently.
The study was posted to the arXiv preprint server on November 19, 2026.
This article has been fact checked for accuracy, with information verified against reputable sources. Learn more about us and our editorial process.
Last reviewed on .
Article history
- Latest version
Reference(s)
- Luo, Jinbin. “TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming.” arXiv.org <http://arxiv.org/abs/2611.26619v1>.
Cite this page:
- Posted by James Whitmore