Hidden Limits of Biological AI Models Reveal Where Predictions Fail

Artificial intelligence models trained on millions of biological measurements are transforming research, but a new review reveals that their outputs often lack clear reliability limits. These powerful computational tools can predict protein structures and cellular responses, yet determining when their predictions can actually be trusted remains a major hurdle for scientists.

Biological foundation models operate by compressing vast amounts of raw data into dense numerical representations called embeddings. Unlike human language, which possesses a structured and agreed-upon grammar, biological data represents indirect measurements shaped by experimental design and assay technology. Consequently, these models often learn statistical patterns and sampling histories rather than underlying physical mechanisms.

Defining the Boundaries of Computational Knowledge

To address this uncertainty, researchers from St. Jude Children’s Research Hospital and other institutions have proposed framing model reliability through a concept known as the knowledge boundary. As detailed in a study available on the arXiv preprint server, five core design elements dictate where these artificial intelligence systems succeed or fail. These elements include the input data and tokenization strategy, model architecture, training scheme, evaluation framework, and post-analysis methods.

For instance, training objectives that rely on masked language modeling or next-token prediction reward models for identifying what occurs together in data rather than uncovering causal relationships. Independent benchmarks frequently demonstrate that simple linear models or training data averages can match or exceed the performance of complex foundation models when predicting genetic perturbation responses.

The Role of Autonomous Agents in Modern Research

Biologists are increasingly turning to agentic systems, which are automated assistants built on large language models that can plan multistep experiments and call foundation models as tools. However, these autonomous agents can inherit the blind spots of the underlying models they utilize. Researchers noted instances where an agent reached a correct gene conclusion through reasoning that actively contradicted both its own tool output and experimental evidence.

To prevent silent errors from propagating through automated workflows, the study authors recommend releasing every foundation model with a standardized, machine-readable boundary statement. Such transparency would detail training species, context limits, and known failure modes, allowing human scientists and automated agents to determine whether a specific biological query falls safely within a model domain.

The study was posted to the arXiv preprint server on November 12, 2026.