View source for Joint embedding predictive architecture

{{Short description|Self-supervised neural network design that learns by predicting the embedding of one part of an input from another}}
{{About|the machine-learning architecture|the company's essay applying it to ionic current|The Nanopore World Model}}
A '''joint embedding predictive architecture''' ('''JEPA''') is a design for self-supervised learning in which a network learns from unlabeled data by predicting the embedding of one part of an input from the embedding of another part. The loss is computed between representations, not between pixels or samples, so details that cannot be predicted are free to drop out of the representation. Yann LeCun proposed the idea in 2022 as a component of a world model for autonomous agents.<ref name="lecun">{{cite web |url=https://openreview.net/forum?id=BZ5a1r-kVsf |title=A path towards autonomous machine intelligence, version 0.9.2 |author=LeCun, Yann |website=OpenReview |date=June 27, 2022}}</ref> Researchers at [[Meta AI]] built I-JEPA for images in 2023, V-JEPA for video in 2024 and V-JEPA 2 in 2025, and the recipe has been tried on electrocardiograms and other signals.

A JEPA is easy to break. If the encoder and the predictor return the same vector for every input, the prediction error is zero and nothing has been learned. Preventing this collapse is the main engineering problem, and a collapsed model reports a loss that looks excellent.

== Origins ==
Joint embedding is older than the name. Siamese networks, in which two copies of one network with shared weights embed two inputs so that the distance between the embeddings says whether the inputs match, were used in 1993 to verify signatures.<ref name="bromley">{{cite journal |last1=Bromley |first1=J. |last2=et al. |title=Signature verification using a "Siamese" time delay neural network |journal=International Journal of Pattern Recognition and Artificial Intelligence |volume=7 |issue=4 |pages=669-688 |year=1993 |doi=10.1142/S0218001493000339}}</ref> LeCun's position paper of 2022 had a different aim. It argued that an agent needs a model of the world that can predict the consequences of actions, and that such a model should predict abstract representations, so that the details of the world that cannot be predicted can be left out. It proposed a hierarchy of such predictors under the name H-JEPA, and it reported no experiments; they began with I-JEPA the following year.<ref name="lecun" />

== Architecture ==
[[File:JEPA_architecture.svg|thumb|right|upright=1.4|alt=A diagram of a joint embedding predictive architecture, with context and target encoders, a predictor and a loss between embeddings, above a masked autoencoder with a decoder and a loss in signal space|A joint embedding predictive architecture compared with a masked autoencoder, drawn for a signal in time. The JEPA predicts the embedding of the hidden stretch and compares it with the target encoder's embedding; the autoencoder reconstructs the hidden samples. The drawing is schematic.]]
A JEPA has three trained parts. The context encoder turns the visible part of an input into embeddings, and the target encoder does the same for the hidden part to give the answer. Given the context embeddings and the position of the hidden part, the predictor outputs a guess at the target embeddings. Training minimizes the distance between guess and answer, usually as a squared error, and no decoder ever maps embeddings back to the input:

<math display="block">\mathcal{L} = \bigl\lVert\, g_\phi\bigl(f_\theta(x_\mathrm{context}),\, p\bigr) - \mathrm{sg}\bigl(f_{\bar\theta}(x_\mathrm{target})\bigr) \bigr\rVert^2,</math>

where ''f'' is the encoder, ''g'' the predictor, ''p'' the position of the target and sg the stop-gradient, which blocks learning signals from flowing into the target. In I-JEPA the target encoder's weights are an exponential moving average of the context encoder's, so the answer changes slowly.

I-JEPA predicts the representations of several target blocks of an image from a single context block. Its authors found the masking strategy to be the main design choice: the target blocks must be large enough to be semantic, and the context block informative enough, which means spread across the image. With these choices a ViT-Huge/14 trained on ImageNet with 16 A100 GPUs in under 72 hours, using no hand-crafted data augmentations.<ref name="ijepa">{{cite journal |last1=Assran |first1=M. |last2=et al. |title=Self-supervised learning from images with a joint-embedding predictive architecture |journal=arXiv |year=2023 |doi=10.48550/arXiv.2301.08243}}</ref> V-JEPA applied the method to video as the sole objective, with no pretrained image encoder, text, negative examples or reconstruction.<ref name="vjepa">{{cite journal |last1=Bardes |first1=A. |last2=et al. |title=Revisiting feature prediction for learning visual representations from video |journal=arXiv |year=2024 |doi=10.48550/arXiv.2404.08471}}</ref>

Two neighboring families mark the boundaries. A masked autoencoder hides patches of an image and reconstructs the missing pixels, so the model must account for every detail, predictable or not.<ref name="mae">{{cite journal |last1=He |first1=K. |last2=et al. |title=Masked autoencoders are scalable vision learners |journal=arXiv |year=2021 |doi=10.48550/arXiv.2111.06377}}</ref> A contrastive method such as SimCLR pulls together the embeddings of two views of one image and pushes apart those of different images.<ref name="simclr">{{cite journal |last1=Chen |first1=T. |last2=et al. |title=A simple framework for contrastive learning of visual representations |journal=arXiv |year=2020 |doi=10.48550/arXiv.2002.05709}}</ref> What a JEPA takes from them is the embedding space of the second and the masking of the first; what it predicts is the embedding of the hidden part.

== Collapse ==
Joint-embedding methods share a trivial solution in which the encoder outputs a constant, and each family prevents it in its own way. Contrastive methods use negative pairs. BYOL trains an online network to predict a slowly updated copy of itself and needs no negatives.<ref name="byol">{{cite journal |last1=Grill |first1=J.-B. |last2=et al. |title=Bootstrap your own latent: a new approach to self-supervised learning |journal=arXiv |year=2020 |doi=10.48550/arXiv.2006.07733}}</ref> VICReg adds an explicit penalty on the variance of the embeddings along each dimension, and a term that decorrelates the dimensions.<ref name="vicreg">{{cite journal |last1=Bardes |first1=A. |last2=Ponce |first2=J. |last3=LeCun |first3=Y. |title=VICReg: variance-invariance-covariance regularization for self-supervised learning |journal=arXiv |year=2021 |doi=10.48550/arXiv.2105.04906}}</ref> Total collapse is one failure among several. In dimensional collapse the embeddings span a smaller space than the one available, and it occurs in contrastive learning too.<ref name="jing">{{cite journal |last1=Jing |first1=L. |last2=et al. |title=Understanding dimensional collapse in contrastive self-supervised learning |journal=arXiv |year=2021 |doi=10.48550/arXiv.2110.09348}}</ref>

The loss does not reveal collapse. The authors of a 2022 study of joint-embedding methods point out that, with no input reconstruction to look at, there are no "visual cues of unsuccessful training", and that the loss values are "non informative". Their remedy is the effective rank of the embeddings, which can be computed without labels and tracks downstream performance.<ref name="rankme">{{cite journal |last1=Garrido |first1=Q. |last2=et al. |title=RankMe: assessing the downstream performance of pretrained self-supervised representations by their rank |journal=arXiv |year=2022 |doi=10.48550/arXiv.2210.02885}}</ref>

== Family ==
{| class="wikitable"
|+ Models in the JEPA family
! Year !! Model !! Data !! Result
|-
| 2023 || I-JEPA || ImageNet images || A ViT-Huge/14 trained in under 72 hours on 16 A100 GPUs<ref name="ijepa" />
|-
| 2024 || V-JEPA || 2 million videos || With a frozen backbone, 81.9% on Kinetics-400, 72.2% on Something-Something v2 and 77.9% on ImageNet-1K<ref name="vjepa" />
|-
| 2024 || ECG-JEPA || Ten public ECG databases, over 1 million records || An AUC of 0.945 on the PTB-XL all-statements task, ahead of invariance-based and generative pretraining<ref name="ecg">{{cite journal |last1=Weimann |first1=K. |last2=Conrad |first2=T. O. F. |title=Self-supervised pre-training with joint-embedding predictive architecture boosts ECG classification performance |journal=arXiv |year=2024 |doi=10.48550/arXiv.2410.13867}}</ref>
|-
| 2025 || TS-JEPA || Standard time-series datasets || Matched or passed baselines on classification and forecasting<ref name="tsjepa">{{cite journal |last1=Ennadir |first1=S. |last2=et al. |title=Joint embeddings go temporal |journal=arXiv |year=2025 |doi=10.48550/arXiv.2509.25449}}</ref>
|-
| 2025 || V-JEPA 2 || Over 1 million hours of video || 77.3% on Something-Something v2; a version post-trained on under 62 hours of unlabeled robot video planned pick-and-place tasks zero-shot on robot arms in two laboratories<ref name="vjepa2">{{cite journal |last1=Assran |first1=M. |last2=et al. |title=V-JEPA 2: self-supervised video models enable understanding, prediction and planning |journal=arXiv |year=2025 |doi=10.48550/arXiv.2506.09985}}</ref>
|}

LeCun's company, [[AMI Labs]], is built around world models of the kind the 2022 paper described.<ref name="ami">{{cite news |title=Who's behind AMI Labs, Yann LeCun's 'world model' startup |work=TechCrunch |author=Anna Heim |date=January 23, 2026 |url=https://techcrunch.com/2026/01/23/whos-behind-ami-labs-yann-lecuns-world-model-startup/}}</ref>

== For signals in time ==
The argument for using a JEPA on a noisy recording is that noise defeats reconstruction. The TS-JEPA authors begin from it: methods that reproduce masked parts of the input, they write, "can be vulnerable to the presence of noise or confounding variables", and they move the task to latent space.<ref name="tsjepa" /> White noise cannot be predicted, so a model that predicts samples carries an error floor equal to the noise variance whatever its size, and for Gaussian noise of 12.8 pA rms, the thermal noise of a 1 MΩ resistor in a band of 10 kHz, that floor is 164 pA<sup>2</sup> per sample (see [[Johnson-Nyquist noise]]). A JEPA is not asked to reach it, because its embedding may leave the noise out.

== At Molecular Reality ==
{{Main|The Nanopore World Model}}
The company's plan to train a JEPA on the ionic current through a [[nanopore]] is set out in an interactive essay of 2026 in the voice of [[Keltar]]. The essay calls the decision to predict in a learned space "the single best decision in this whole stack", because "A 200 kHz current trace is mostly thermal noise, capacitive ringing, and the amplifier's Bessel filter talking to itself." It warns that video JEPAs come with a slowly updated teacher network, that a home-made version for current traces "does not get that for free", and that without one "the predictor collapses every signal to one constant vector and reports a beautiful, meaningless loss of ~0."<ref name="wm">{{cite web |url=https://molecularreality.com/nanopore-world-model |title=Keltar's Critique v2: The Nanopore World Model (Hardened) |publisher=Molecular Reality Corporation |date=June 2026}}</ref>

Around the JEPA, the plans put a Conformer for the current, a Perceiver to fuse current, voltage and microscope video, and a language model to narrate the results.<ref name="records">{{cite web |title=Company records |publisher=Molecular Reality Corporation |date=October 2026}}</ref> Before any of it is built, the essay proposes a test: pretrain on public nanopore recordings and check whether the frozen embeddings beat five classical numbers, "for ~$50 of GPU". The company's guide files the resulting [[world model]] of the signal under plans: "The model itself is still ahead of us."<ref name="faq">{{cite web |url=https://molecularreality.com/FAQ |title=Keltar's Field Guide to Molecular Reality |publisher=Molecular Reality Corporation |date=August 2026}}</ref> The wider program is described in [[Machine learning at Molecular Reality]] and [[Machine learning on nanopore signals]].

== See also ==
* [[World model]]
* [[The Nanopore World Model]]
* [[Meta AI]]
* [[AMI Labs]]
* [[Conformer (neural network)]]
* [[Mamba (deep learning)]]

== References ==
{{Reflist}}

{{Portal bar|Molecular Reality}}

[[Category:Machine learning]]
[[Category:Artificial neural networks]]
[[Category:Unsupervised learning]]