Predicting dynamic expression patterns in budding yeast with a fungal DNA language model

Reviewer #1 (Public review):

Summary:

The authors ask whether self-supervised pretraining on related fungal genomes gives a useful prior for predicting gene expression in S. cerevisiae, where the compact ~12 Mb genome supplies too few independent windows to train a large supervised model from scratch. They pretrain a BERT-style masked DNA language model on a corpus of fungal genomes and fine-tune it to predict RNA-seq gene expression. They found a surprisingly (to me) large improvement in performance: 0.78 prediction Pearson R versus 0.67 for a randomly initialized model.

The manuscript also presents a new experimental resource: 3,053 RNA-seq data sets with perturbations using the YETI experimental platform to upregulate specific genes.

Strengths:

(1) Overall, the manuscript is well-written and is likely to be impactful.

(2) The protocol handles the train/test split of orthologous sequences well, which is nontrivial.

(3) The authors did a good job "steelman-ing" Shorkie_Random_Init: it received its own learning-rate sweep, and the authors tested two reduced-capacity from-scratch architectures to rule out some overparameterization issues.

(4) The public codebase is unusually well-organized.

Weaknesses:

I have a number of comments, none of which significantly impact the main findings.

Two analyses appear missing from the MPRA section: (1) Does self-supervised pre-training improve MPRA models such as DREAM-RNN? and (2) Is Shorkie better than Shorkie_Random_Init at the MPRA task? The language about "correlative, non-causal" associations makes me think the authors tried this and got poor results; it would be informative to include these as a supplementary negative result. (There is also (3): Does MPRA pre-training improve genomic models? But this is clearly out of scope for this paper.)

Reviewer #2 (Public review):

Summary:

Chao et al. study whether multi-species self-supervised pretraining learns sequence representations that can be transferred to improve downstream expression prediction in budding yeast. The rationale is that the budding yeast genome is relatively small and compact, which may not provide enough sequence variation for standard reference-genome-based supervised learning.

This paper has two stages. They first train U-Net-style DNA language models on collections of fungal genomes with increasing phylogenetic breadth using a BERT-style masked language modeling (MLM) objective. The model with the best sequence reconstruction (i.e., lowest perplexity) on held-out budding yeast sequences is called Shorkie LM. They then use Shorkie LM's weights to initialize Shorkie, a supervised model for predicting functional genomic tracks (RNA-seq, ChIP-exo, ChIP-MNase) in budding yeast. Compared with a randomly initialized model with the same architecture, Shorkie clearly performs better on held-out expression prediction and outperforms the randomly initialized model on all three variant effect benchmarks. This is good evidence that the pretraining is beneficial. For Shorkie LM and Shorkie, the authors also use interpretation methods to analyze motifs captured across a range of loci, and the results seem to be consistent with known yeast biology.

Overall, the experiments are well thought out and controlled, and most claims are supported with sufficient evidence. While the idea that pretraining can be beneficial for budding yeast has been reported previously (for scalar expression prediction) [1], this paper's exploration of optimal evolutionary scope and genomic language model pretraining in general still holds practical value for the field. The Shorkie model itself is also a useful resource and ranks first on many tasks in a recent public benchmark for fungal sequence-to-expression models [2]. I only have some minor concerns/suggestions.

Strengths:

(1) The authors put considerable effort into evaluating their modeling choices. For the pretraining corpora, they compare four phylogenetic scopes (a species, strain, order, kingdom). Each corpus is evaluated across multiple model architectures/capacities. Extra care is taken with homology filtering between training and held-out data. For supervised transfer, several candidate language models are carried forward to test whether their ranking by language model selection criterion (perplexity) predicts their ranking on the downstream task. In addition, the same-architecture randomly initialized baseline for Shorkie is itself optimized for learning rate, with additional randomly initialized CNN and U-Net models included as additional controls, giving some strong nulls to compare against.

(2) The experimental dataset generated in this study is also a valuable resource. Using the ministat array, the authors generated ~3,000 time-resolved RNA-seq profiles following transcriptional regulator inductions. Such data are important for understanding transient regulatory events and how they shape the transcriptome over time.

(3) Several figures include schematic panels that make the whole workflow easy to understand.

Weaknesses:

(1) As acknowledged by the authors, the current evolutionary "sweet spot" at 165 Saccharomycetales genomes is potentially confounded by factors like corpus size, annotation quality, and optimization difficulties. I agree that it would be hard to disentangle these factors, but a relatively simple control appears to be missing. Although unlikely, it is still possible that the "sweet spot" is driven partly by the amount of data rather than the phylogenetic scope. One simple control would be to sample 165 genomes from the broader fungal corpus and match the total number of training windows to the Saccharomycetales dataset.

(2) L148, "These results demonstrate that Shorkie LM captures conserved regulatory grammar." The evidence presented in this section is mainly about recovery of known fungal/yeast sequence motifs, which are more like words than grammar. The latter is usually understood as relationships between motifs such as their spacing, multiplicity, and arrangement. So I suggest the authors either tone down the claim or provide additional analyses that directly test this. One possibility would be to generate nucleotide dependency maps [3] for a handful of loci with well-characterized regulatory syntax.

(3) Figure 2E. The interpretation of t-SNE clusters might be confounded by the length of genomic elements. The five classes of elements shown in the plot have very different length distributions, but they are extracted and zero-padded to the same input length, and their embeddings are mean-pooled across the full padded sequence. As a result, the fraction of real sequence entering the mean embedding differs substantially across genomic element classes, which could contribute to observed separation independently of learned regulatory features.

(4) Lines 216-218, "transfer learning from pan-fungal self-supervised pretraining yields generalizable representations of exon-intron structure and regulatory grammar, substantially improving in expression prediction." Similar to point 2, I think this statement is somewhat stronger than what the current evidence supports. The results clearly show that pretraining improves downstream expression prediction, but it is less clear that this improvement can be specifically attributed to the transfer of regulatory grammar and gene structure, rather than motif representations or more general learned inductive biases acquired during pretraining. This is itself an interesting question. It might be possible to get at it by selectively reinitializing the convolution layers vs the transformer blocks while keeping the rest of the Shorkie LM weights, although this could still be hard to interpret if the relevant information is distributed across the model.

(5) The ISM for Figures 4 and 5 seems to correspond to the summed predicted coverage across all output bins (Methods, Equation 17). I was unclear about the intended goal here. For example, when mutating the ATG42 promoter in Figure 5, is the goal to quantify the predicted effect on ATG42 expression specifically? If so, should the ISM instead be computed by summing only the bins covering the target gene? Otherwise, predicted coverage from neighboring genes could also contribute to the ISM score and affect the comparison across induction time points.

References:

[1] Wang Y, Cai Z, Zeng Q, Gao Y, Ouyang J, Xu Y, et al. Genomic Touchstone: benchmarking genomic language models in the context of the central dogma. bioRxiv. Preprint posted online June 30, 2025. doi:10.1101/2025.06.25.661622

[2] Schneider T. ybench: a benchmark for fungal sequence-to-expression models. GitHub. Accessed August 18, 2026. https://github.com/Tom-Ellis-Lab/yeast-seq2expression-benchmark

[3] Tomaz da Silva P, Karollus A, Hingerl J, et al. Nucleotide dependency analysis of genomic language models detects functional elements. Nat Genet. 2025;57(10):2589-2602. doi:10.1038/s41588-025-02347-3