1997 Pentium II Powers Llama AI on 128 MB Windows 98 PC

A team at EXO Labs managed to coax a decade‑old Windows 98 PC into generating text with modern Llama language models. The machine, purchased on eBay for £118.88, ran on a Pentium II processor with 128 MB of RAM and a 1.6 GB hard drive. After a series of hardware hacks and software adaptations, the system produced up to 39.31 tokens per second with a 260 k‑parameter model, all without a graphics card.

Bringing Legacy Peripherals Online

Initial attempts to use USB keyboards and mice failed, forcing the researchers to rely on PS/2 devices that required a specific connection order. Rewritable disks were ignored by the OS, and a 4 TB USB stick exceeded the FAT32 limits. To enable data transfer, the PC was linked via Ethernet to an M4 MacBook Pro running a FileZilla FTP server. Assigning static IP addresses yielded sub‑millisecond ping times, but the team only achieved successful file transfers after switching the FTP client to binary mode.

Porting a Minimal Llama Inference Engine

EXO Labs based their effort on Andrej Karpathy’s llama2.c, a single‑file C implementation designed for educational purposes rather than full‑scale deployment. Because the original code relied on modern language features absent from the era‑specific Borland C++ 5.02 compiler, the team rewrote variable declarations, swapped long long for a custom DLONGWORD type, and replaced clock_gettime with Windows’ GetTickCount(). Memory‑mapped model loading caused segmentation faults, so the loading routine was simplified. The resulting executable, dubbed llama98.c, could run directly under Windows 98.

Benchmark Results on a 1990s Processor

With the port functional, the team measured inference speeds on the legacy hardware. A 260 k‑parameter Llama‑style model achieved 39.31 tokens per second, while a 15‑million‑parameter model managed just over 1 token per second. By contrast, Karpathy reported roughly 110 tokens per second for the same 15‑M model on an M1 MacBook Air, highlighting the dramatic performance gap between modern silicon and the 1998 platform.

EXO also provided a speculative figure for a 1‑billion‑parameter Llama 3.2 model—0.0093 tokens per second—derived from a combination of a shrunken model shard that fit into memory and disk‑read throughput measurements. Full execution of the billion‑parameter checkpoint was not performed.

Karpathy’s original code stores weights as 32‑bit floats, meaning a 7‑billion‑parameter checkpoint would occupy about 26 GB. An int8 quantized variant reduces the size to 6.7 GB, improving performance on Karpathy’s Linux test rig from 4.6 to 14 tokens per second, albeit with a modest loss in output quality. These quantization results pertain to the llama2.c repository rather than the Pentium II experiment.

Exploring More Compact Model Architectures

Looking ahead, EXO Labs discussed BitNet, a technique that restricts weight values to three possibilities (‑1, 0, 1), effectively using about 1.58 bits per weight. This ternary representation turns most multiplications into simple additions or subtractions, promising reductions in both storage and compute requirements. A 7‑billion‑parameter BitNet model would need roughly 1.38 GB, a size that could theoretically fit on the PC’s 1.6 GB drive.

EXO cited performance numbers for BitNet on contemporary hardware: 52 tokens per second on an M2 Ultra and 18 tokens per second on an Intel i7. They also projected 5–7 tokens per second for a 100‑billion‑parameter model running on a single CPU. Although these figures stem from recent experiments on modern processors, they illustrate a potential pathway for running larger models on constrained devices.

EXO announced the release of MLX‑BitNet in April 2024 for Apple Silicon and highlighted a BitNet implementation for protein‑language modeling presented at ICML 2024. At the time of the Windows 98 demonstration, no open‑source large‑scale BitNet model was publicly available, and the team planned a full‑size training run in 2025.

Byte‑Level Models Simulating a Simplified CPU

A separate study, Beyond Language Models: Byte Models are Digital World Simulators, investigated whether a model trained directly on raw bytes could learn to predict processor states. Researchers Shangda Wu, Xu Tan, Zili Wang, Rui Wang, Xiaobing Li and Maosong Sun introduced bGPT, a 110‑million‑parameter model evaluated on tasks ranging from text generation to CPU‑state forecasting.

For the CPU experiment, the authors generated 2.1 million training examples from a toy CPU simulator. Each example paired a 1 KB memory segment containing a sequence of instructions with a 16‑byte snapshot of the register file after execution. The instruction set comprised 21 types and 43 variants, covering data movement, logical, and arithmetic operations. Programs varied from one to 256 randomly generated instructions, deliberately excluding jumps to avoid infinite loops; every program terminated with a halt instruction. This design yielded over 516 million possible scenarios, and an evaluation set of 21 000 examples was held out.

Training accuracy rose sharply with dataset size. With 10 000 examples, the bGPT4 configuration reached 27.24 % byte‑wise accuracy. Scaling to 100 000 examples improved bGPT5 to 99.97 %**, and a full million‑example run pushed bGPT6 to 99.996 %**. The authors noted an abrupt performance jump around the 11th epoch for bGPT5, suggesting a possible emergent capability, while also acknowledging alternative explanations such as metric non‑linearity or overfitting.

The authors cautioned that the findings apply only to the simplified CPU used in the study; real‑world processors are considerably more complex than the simulated model.