NVIDIA releases weights for JEPA-DNA, a DNA model trained to predict representations of entire regions

NVIDIA releases weights for JEPA-DNA, a DNA model trained to predict representations of entire regions

On 16 July, NVIDIA released the weights, code, and reproducibility settings for JEPA-DNA. The method adds a second task to the standard training of genomic models: reconstructing the internal numerical representation of an entire sequence from a visible fragment.

A genomic model reads DNA as a string of four letters. It usually learns to reconstruct masked letters or predict the next one. However, the function of a genomic region often depends on a combination of elements distributed across a long sequence. A promoter tells the cell where to begin transcribing a gene, while a splice site helps assemble RNA.

In the JEPA-DNA preprint, first published on 19 February, the authors retained the nucleotide prediction task and added a second objective. One encoder receives a sequence containing masked regions. Another sees the complete sequence and produces its internal numerical representation. A predictor learns to recover this representation from the available context. The authors propose that this objective may help the model capture the function of a region alongside information about neighboring nucleotides.

The evaluation covered three base models: DNABERT-2, Nucleotide Transformer v3 and HyenaDNA. Nine tasks measured representation quality after training a simple classifier, while another eight evaluated DNA variants by comparing the representations of the original and altered sequences. For HyenaDNA, AUROC in the promoter recognition task increased from 0,686 to 0,763. In the transcription factor binding task, the same metric decreased from 0,698 to 0,638. The same backbone therefore produces different results across tasks. The value of the JEPA objective must be tested for each specific application and cannot be inferred from the average improvement alone.

NVIDIA released the HyenaDNA weights, and the repository provides instructions for three sets of weights and for running GFMBench evaluations. This benchmark tests DNA models on promoters, splicing, and the effects of genetic variants. The weights are available under a noncommercial license.

A laboratory can compare the original HyenaDNA with the JEPA-DNA version on its own DNA sequence task. The public weights and reproducibility settings make it possible to evaluate the method beyond the results reported in the preprint tables.