← back to the JEPA explainer the catch · collapse

The model doesn’t give a vague answer.
It stops answering.

This is the one part that trips everybody, because the obvious guess is wrong in an interesting way. Your guess was that self-scoring produces a safe average. Half right — let’s do the half that isn’t.

Your instinct, marked

You had the cause right and the symptom wrong.

“Because it is scoring itself, it’ll give an average answer which works for all.”

✓ “Because it is scoring itself”

Exactly the right diagnosis. Self-scoring is the vulnerability, and spotting that is the hard part. Everything below is just working out what the model does with the loophole you found.

✕ “It’ll give an average answer”

This is a real failure — it’s just a different one, belonging to the other approach. An average is a blur: still describing the picture, badly. Collapse isn’t a bad description. It’s the absence of one.

The two get confused constantly, so it’s worth separating them properly before going on.

The distinction

Blur is hedging. Collapse is quitting.

BLUR · what you guessed the answer is a mush of all the real answers dog runs left dog runs right dog sits still a smear of all three Useless as a prediction, but not empty — the smear still knows it’s about a dog. The model tried and hedged. It cannot do worse than this. The thing it is being marked against is the actual photo, and it has no way to change the actual photo. COLLAPSE · what actually happens the answers are replaced with one meaningless constant dog runs left a cup falling a door opening 7 the same code. every time. for every input, forever. Nothing about a dog survives. Nothing about anything survives. The code carries zero information. And its score is perfect. Predicted gist: 7. Encoder’s gist: 7. Distance: zero. There is no better score available anywhere. Blur is the best it can manage. Collapse is it discovering it never had to try.

The mechanism

Who writes the right answer?

This is the whole thing. In ordinary training, the right answer comes from outside the model — a human label, or the actual pixels of the actual photo. In JEPA, the right answer is produced by the model itself.

GENERATIVE — TARGET COMES FROM REALITY model its guess the actual photo a fact. untouchable. compare The model can only change one side of this. Cheating is impossible; hedging is all it has. JEPA — TARGET COMES FROM THE MODEL model its guess the “right answer” not a fact — the model makes it compare Now it owns both sides. It can stop getting better at the question and instead make the question easier. That is the loophole.
You spotted this without being told — “because it is scoring itself”. This diagram is just that sentence, drawn.

It is an exam where the student also writes the paper. They don’t get better at the questions. They make every question “write the letter A.”

Concretely

Watch it happen in numbers.

Three different photos. The gist is a short list of numbers. Here’s an encoder doing its job, and the same encoder having given up.

A working encoder

photoencoder says, visible halfencoder says, hidden halfpredictor must
dog running[0.9, 0.1, 0.4][0.8, 0.2, 0.5]work out where the dog went
cup falling[0.2, 0.7, 0.1][0.3, 0.9, 0.1]work out that it lands
door opening[0.5, 0.3, 0.9][0.4, 0.3, 0.8]work out how far it swung

Different photos, different numbers. The predictor has to genuinely understand something to get close. It never gets to zero error — and that’s fine, that’s what learning looks like.

The same encoder, collapsed

photoencoder says, visible halfencoder says, hidden halfpredictor must
dog running[7, 7, 7][7, 7, 7]say “7, 7, 7”
cup falling[7, 7, 7][7, 7, 7]say “7, 7, 7”
door opening[7, 7, 7][7, 7, 7]say “7, 7, 7”

Error: exactly zero. On every example. Forever. The training process is a hill-climber looking for low error, and this is the bottom of the valley. Left alone, it will find it — not because it’s clever, but because gradient descent has no concept of cheating. It goes downhill.

And notice: nothing here is an average. [7,7,7] isn’t the mean of the three varied gists above. Those gists no longer exist anywhere — the encoder stopped producing them.

The sticking point

“But surely the real gist is something else?”

This is where everyone stops, and it’s the right place to stop. The answer is hiding in the word real — which is a word I shouldn’t have used, so let’s take it apart.

The gist does not live in the photo. The gist is whatever the encoder says.

There is no fact of the matter about what the gist of a photo is. Nobody wrote it down. It isn’t recoverable from the image. There is only one way to obtain it: run the encoder on the hidden half and see what numbers come out. Those numbers are the gist, by definition.

So when the encoder has become “say 7 to everything,” the gist of the hidden half is 7. Not approximately 7. Not a wrong answer that should have been something else. Exactly 7, because “the gist” is defined as the encoder’s output, and that is what the encoder outputs.

Contrast that with the generative setup, where the target is the actual photo. The photo is a fact. You cannot define your way out of it. That difference — target is a fact versus target is something the model manufactures — is the entire reason one of these can collapse and the other cannot.

So what did it change the question to?

Nothing about the wording. The question is still, word for word: what will the encoder say about the hidden half?

Here is the part that is easy to miss:

The question was never about the photo. It was always about the encoder. You only thought it was about the photo because, normally, what the encoder says depends on the photo.

Try it as a question about a person. I ask you: “what will Ben say when he sees the right half of this photo?” That is a question about Ben. To answer it well you do have to understand photographs — because Ben’s answers track what is in them. Predicting Ben is a good exercise.

Unless Ben has quietly decided to say “Blue” to everything. Then predicting Ben is trivial, you are perfect at it, and you never learn anything about a photograph again.

The encoder is Ben. You are never predicting the photo. You are predicting what your own encoder says about the photo — and the model owns the encoder.

And one more thing to flip round

It is tempting to read collapse as “every answer is wrong.” It is the opposite: every answer is right. That is why it survives. Training only ever pushes down wrongness — if the answers were wrong, training would repair them. Collapse persists precisely because it scores perfectly. It is the bottom of the valley, and gradient descent rolls downhill.

If the numbers don’t land

Two people describing photos down a phone line.

Anna sees the left half of a photo and describes it. Ben sees the right half. Ben’s job is to guess what Anna will say before she says it. They’re scored on how well Ben’s guess matches Anna’s description. That is JEPA.

Doing it properly

AnnaLeft side, a dog mid-stride, heading right, blurred paws.
Ben guessed“Something moving fast, heading right.” — close. Not perfect, but he clearly understood the photo.

The cheat

Before the game starts, they agree on something.

AnnaBlue.
BenBlue.
AnnaBlue.
BenBlue.

They agree perfectly on every photo for the rest of time. Their score is flawless. Neither of them ever looked at a photo again.

That’s collapse. And notice it is not Ben giving a safe average answer like “some kind of animal, probably.” A safe average would still mean Anna described something. Here, Anna stopped describing. The information was never sent.

Now compare with the generative version of this game: Ben has to redraw the right half of the photo and is scored against the actual photo. He and Anna can agree on whatever they like — the photo is still the photo. The best they can do when unsure is a vague sketch. That’s the blur.

The defences

So how do you stop it?

Two families. The second is what JEPA actually uses, and it’s a genuinely odd trick.

1 · Show it wrong answers too

Alongside every matching pair, feed it mismatched ones — the left of this photo, the right of that one — and demand those be scored far apart.

Now “everything is 7” fails instantly: if all codes are identical, mismatched pairs are close too, and the model is punished. Straightforward, and it works.

The cost: as the code gets richer, the number of wrong examples you need to pin it down explodes. It stops being practical exactly when you want the codes to be detailed.

2 · Make the right answer keep moving

Don’t use one encoder for both sides. Use two — and make the one that produces the right answer a slow-motion copy of the one that’s learning.

The student sprints; the answer key strolls along behind it, always a few steps back. To collapse, both have to be sitting on the same constant at the same moment — but every time the student moves toward it, the answer key is still back where the student was a moment ago, and the error goes up, not down. The shortcut stops paying.

This is what I-JEPA and V-JEPA use. BYOL and data2vec use it too.

WHY A MOVING TARGET HELPS step 1step 2step 3 step 4step 5 the student — moves every step the answer key — trails behind The gap never closes to nothing. There is always real work left, so “stand still and agree” is never the cheapest move.
There’s also a third family that simply adds a penalty for the codes being too alike — forcing variety directly. VICReg and friends take that route.

The honest footnote

Nobody fully agrees on why the lagging-copy trick works. It reliably does, across many labs and architectures, and there are several competing explanations — but it was found by experiment, not derived from theory. If it feels slightly like a magic trick, that’s not you misunderstanding it. It is a bit of a magic trick.

What LeCun actually prescribes

Never optimise predictability on its own.

He saw this hole before anyone built the thing. The position paper doesn’t list one objective, it lists four — and two of them exist purely to stop the model taking the shortcut. The fix isn’t a patch applied afterwards; it’s in the specification.

COLLAPSE the useful model NOISE perfectly predictable, says nothing compressed enough to predict, rich enough to mean something keeps everything, predicts nothing “be predictable” pulls this way “be informative” pulls this way Optimise either one alone and you slide off an end. The model you want only exists in the tension.

The four criteria, in plain English

#what it asks forwhy it’s there
1the gist of the visible half must carry real information about itstops collapse
2the gist of the hidden half must carry real information about itstops collapse
3one gist must be predictable from the otherthis is the actual learning
4keep the randomness dial smallstops a second collapse — see below

Criterion 3 on its own is the thing that gets gamed. One and two are the guards. Notice they are not “detect cheating and punish it” — they are a positive requirement that the numbers mean something, which cheating happens to fail.

His stated preference, and why

Contrastive — workable, doesn’t scale

Push mismatched pairs apart. It works. But the number of wrong examples you need grows exponentially with how detailed the gist is — so it fails exactly when you want rich representations. That’s LeCun’s stated objection, not a vague preference.

Regularised — his recommendation

Don’t generate wrong answers at all. Add a term that directly demands the gists vary and don’t duplicate each other. No curse of dimensionality. VICReg, from his own group, is the worked example; the lagging-copy trick in I-JEPA and V-JEPA is the pragmatic cousin.

The second door

Worth knowing, because it catches people who’ve only learned about the first one. The randomness dial — the latent variable — exists so that one input can produce several plausible futures. Give it too much capacity and the model can produce any output from any input.

Then everything is plausible, the score is low everywhere, and you are collapsed again — through a completely different door. You didn’t make all the answers identical this time; you made all the answers acceptable. That is what criterion 4 is standing guard over.

Where theory and practice part company

The regularised story is the principled one, and it is the one LeCun argues for. The lagging-copy trick is the one that is actually winning in the benchmarks. Nobody fully agrees on why it works — it was found by experiment and the explanation is still contested. Both facts are true at once, and it is worth holding them together rather than picking the tidier one.

One paragraph

The whole thing, compressed.

JEPA scores itself by comparing its prediction to its own summary of the answer. There is no ground truth here: the “correct” gist is simply whatever the encoder says, so the model controls both sides of the comparison. The cheapest way to score well is therefore not to predict better — it is to make every summary identical, at which point every answer is right, the score is perfect, and the summaries mean nothing. This is collapse. Not a vague answer; the disappearance of the answer. The fix is to make the “correct” gist a slowly-trailing copy of the model, so standing still never wins.