Deep Learning Models Fail to Generalize in Brain-Video Alignment
Biology

Deep Learning Models Fail to Generalize in Brain-Video Alignment

A diagnostic sham control reveals that high cross-subject decoding accuracy can rely on memorization rather than true neural tracking.

By Priya Sharma
Published:
Email this Article
Deep Learning Models Fail to Generalize in Brain-Video Alignment
Architecture of the temporal contrastive model. The EEG branch (orange) applies spatial filtering followed by temporal convolutions to produce a scalar sequence at 8 Hz. The video branch (blue) aggregates pretrained VideoMAE features over the middle transformer layers and projects them to a scalar sequence via a lightweight MLP. Both outputs are zero-meaned and L2L_{2}-normalized such that their inner products become equivalent to a Pearson correlation, which populates the similarity matrix. The model is trained with a symmetric InfoNCE loss. Yuanyuan Yao et al. / arXiv

When human brains process continuous natural videos, cortical neural activity synchronizes with the moving visual stimuli in a process known as neural tracking, allowing researchers to decode a person’s attention from electroencephalography recordings. A team of researchers from KU Leuven investigated whether advanced nonlinear deep learning models could improve this EEG-video alignment compared to traditional linear methods, analyzing data from participants watching naturalistic video content. The study is available on the arXiv preprint server and highlights major hurdles in how brain-computer interface decoders are evaluated.

How Deep Learning Models and Linear Baselines Match Up

The authors paired a lightweight electroencephalography encoder with pretrained representations from VideoMAE, a masked autoencoder architecture that learns spatiotemporal dynamics by reconstructing masked video patches. This two-branch network was trained using a correlation-based InfoNCE objective, optimizing the model to map brain activity and video features into a shared embedding space. They benchmarked this deep contrastive model against a standard linear canonical correlation analysis baseline across three distinct generalization regimes.

Under the leave-one-subject-out regime, where models encounter new participants watching familiar videos, the contrastive model outperformed the linear baseline significantly across 10-second, 20-second, and 30-second windows. However, when tested on unseen temporal segments of familiar videos in a leave-video-segment-out setup, the performance advantage disappeared, with the deep model performing on par with the linear approach. Most strikingly, when forced to extrapolate to entirely novel videos in a leave-one-video-out regime, the deep contrastive model consistently underperformed compared to the linear baseline.

Figure 1: Layer-wise received attention across all 12 layers of VideoMAE for the first 1-second window of an example video. Left: the middle frame of the window, shown for reference. Right: received attention per spatial patch for each layer, averaged over heads, query tokens, and the 8 temporal positions, and upsampled from the 14×1414\times 14 patch grid to the frame resolution. Colors show attention relative to a uniform baseline (1×1\times) on a shared log2\log_{2} scale: blue represents below-uniform attention, and red represents above-uniform attention.
Layer-wise received attention across all 12 layers of VideoMAE for the first 1-second window of an example video. Left: the middle frame of the window, shown for reference. Right: received attention per spatial patch for each layer, averaged over heads, query tokens, and the 8 temporal positions, and upsampled from the 14×1414\times 14 patch grid to the frame resolution. Colors show attention relative to a uniform baseline (1×1\times) on a shared log2\log_{2} scale: blue represents below-uniform attention, and red represents above-uniform attention. Credit: Yuanyuan Yao et al. / arXiv

The Illusion of Alignment Revealed by Sham Controls

To understand why high cross-subject accuracy did not translate to novel video generalization, the authors constructed a clever diagnostic sham dataset. By applying a deterministic circular shift of at least 60 seconds to the video features while leaving the electroencephalography data untouched, they completely destroyed the genuine temporal alignment between the brain responses and the visual stimuli while preserving all other structural properties.

When the deep contrastive model was retrained on this sham dataset, it still achieved high match-mismatch accuracy on seen subjects, performing nearly as well as the model trained on real data. In contrast, the linear canonical correlation analysis baseline collapsed entirely to chance levels on the sham data. This discovery indicates that high accuracy under subject-only cross-validation can be deceptive, reflecting a synthesized or memorized correspondence fit to seen training stimuli rather than physically meaningful neural coupling.

(a): PCA scatter
PCA scatter Credit: Yuanyuan Yao et al. / arXiv

Signal-to-Noise Ratios and Group-Level Averages

Despite the vulnerabilities exposed by the sham controls, further investigation revealed that the model does learn genuine, generalizable coupling. When evaluated against a group-averaged electroencephalography reference that suppresses individual noise by averaging across participants, the video-EEG alignment accuracy jumped dramatically, reaching around 75 percent even on entirely unseen videos. This proves that generalizable coupling exists in the data but is heavily masked in single-subject recordings by measurement noise and individual differences in how people respond to visual scenes.

The authors suggest that future research must prioritize raising the single-trial signal-to-noise ratio, either by extracting stronger stimulus-locked brain responses through expressive video representations or by developing better denoising methods. They conclude that neural tracking models should always be evaluated under rigorous joint holdout protocols that isolate both participants and stimuli simultaneously to avoid overestimating true alignment.

The study was posted to the arXiv preprint server on November 19, 2026.

Fact Checked

This article has been fact checked for accuracy, with information verified against reputable sources. Learn more about us and our editorial process.

Last reviewed on .

Article history

  • Latest version

Reference(s)

  1. Yao, Yuanyuan. “Temporal Contrastive Alignment of EEG and Naturalistic Video Only Beats Linear Decoding On In-Sample Stimuli.” arXiv.org <http://arxiv.org/abs/2611.26258v1>.

Cite this page:

Sharma, Priya. “Deep Learning Models Fail to Generalize in Brain-Video Alignment.” BioScience. BioScience ISSN 2521-5760, 20 November 2026. <https://www.bioscience.com.pk/en/subject/biology/temporal-contrastive-alignment-of-eeg-and-naturalistic-video-only-beats-linear-decoding-on-in-sample-stimuli>. Sharma, P. (2026, November 20). “Deep Learning Models Fail to Generalize in Brain-Video Alignment.” BioScience. ISSN 2521-5760. Retrieved November 20, 2026 from https://www.bioscience.com.pk/en/subject/biology/temporal-contrastive-alignment-of-eeg-and-naturalistic-video-only-beats-linear-decoding-on-in-sample-stimuli Sharma, Priya. “Deep Learning Models Fail to Generalize in Brain-Video Alignment.” BioScience. ISSN 2521-5760. https://www.bioscience.com.pk/en/subject/biology/temporal-contrastive-alignment-of-eeg-and-naturalistic-video-only-beats-linear-decoding-on-in-sample-stimuli (accessed November 20, 2026).
End of the article