This is the one part that trips everybody, because the obvious guess is wrong in an interesting way. Your guess was that self-scoring produces a safe average. Half right — let’s do the half that isn’t.
“Because it is scoring itself, it’ll give an average answer which works for all.”
Exactly the right diagnosis. Self-scoring is the vulnerability, and spotting that is the hard part. Everything below is just working out what the model does with the loophole you found.
This is a real failure — it’s just a different one, belonging to the other approach. An average is a blur: still describing the picture, badly. Collapse isn’t a bad description. It’s the absence of one.
The two get confused constantly, so it’s worth separating them properly before going on.
This is the whole thing. In ordinary training, the right answer comes from outside the model — a human label, or the actual pixels of the actual photo. In JEPA, the right answer is produced by the model itself.
It is an exam where the student also writes the paper. They don’t get better at the questions. They make every question “write the letter A.”
Three different photos. The gist is a short list of numbers. Here’s an encoder doing its job, and the same encoder having given up.
| photo | encoder says, visible half | encoder says, hidden half | predictor must |
|---|---|---|---|
| dog running | [0.9, 0.1, 0.4] | [0.8, 0.2, 0.5] | work out where the dog went |
| cup falling | [0.2, 0.7, 0.1] | [0.3, 0.9, 0.1] | work out that it lands |
| door opening | [0.5, 0.3, 0.9] | [0.4, 0.3, 0.8] | work out how far it swung |
Different photos, different numbers. The predictor has to genuinely understand something to get close. It never gets to zero error — and that’s fine, that’s what learning looks like.
| photo | encoder says, visible half | encoder says, hidden half | predictor must |
|---|---|---|---|
| dog running | [7, 7, 7] | [7, 7, 7] | say “7, 7, 7” |
| cup falling | [7, 7, 7] | [7, 7, 7] | say “7, 7, 7” |
| door opening | [7, 7, 7] | [7, 7, 7] | say “7, 7, 7” |
Error: exactly zero. On every example. Forever. The training process is a hill-climber looking for low error, and this is the bottom of the valley. Left alone, it will find it — not because it’s clever, but because gradient descent has no concept of cheating. It goes downhill.
And notice: nothing here is an average. [7,7,7] isn’t the mean of the three
varied gists above. Those gists no longer exist anywhere — the encoder stopped producing them.
This is where everyone stops, and it’s the right place to stop. The answer is hiding in the word real — which is a word I shouldn’t have used, so let’s take it apart.
The gist does not live in the photo. The gist is whatever the encoder says.
There is no fact of the matter about what the gist of a photo is. Nobody wrote it down. It isn’t recoverable from the image. There is only one way to obtain it: run the encoder on the hidden half and see what numbers come out. Those numbers are the gist, by definition.
So when the encoder has become “say 7 to everything,” the gist of the hidden half is 7. Not approximately 7. Not a wrong answer that should have been something else. Exactly 7, because “the gist” is defined as the encoder’s output, and that is what the encoder outputs.
Contrast that with the generative setup, where the target is the actual photo. The photo is a fact. You cannot define your way out of it. That difference — target is a fact versus target is something the model manufactures — is the entire reason one of these can collapse and the other cannot.
Nothing about the wording. The question is still, word for word: what will the encoder say about the hidden half?
Here is the part that is easy to miss:
The question was never about the photo. It was always about the encoder. You only thought it was about the photo because, normally, what the encoder says depends on the photo.
Try it as a question about a person. I ask you: “what will Ben say when he sees the right half of this photo?” That is a question about Ben. To answer it well you do have to understand photographs — because Ben’s answers track what is in them. Predicting Ben is a good exercise.
Unless Ben has quietly decided to say “Blue” to everything. Then predicting Ben is trivial, you are perfect at it, and you never learn anything about a photograph again.
The encoder is Ben. You are never predicting the photo. You are predicting what your own encoder says about the photo — and the model owns the encoder.
It is tempting to read collapse as “every answer is wrong.” It is the opposite: every answer is right. That is why it survives. Training only ever pushes down wrongness — if the answers were wrong, training would repair them. Collapse persists precisely because it scores perfectly. It is the bottom of the valley, and gradient descent rolls downhill.
Anna sees the left half of a photo and describes it. Ben sees the right half. Ben’s job is to guess what Anna will say before she says it. They’re scored on how well Ben’s guess matches Anna’s description. That is JEPA.
Before the game starts, they agree on something.
They agree perfectly on every photo for the rest of time. Their score is flawless. Neither of them ever looked at a photo again.
That’s collapse. And notice it is not Ben giving a safe average answer like “some kind of animal, probably.” A safe average would still mean Anna described something. Here, Anna stopped describing. The information was never sent.
Now compare with the generative version of this game: Ben has to redraw the right half of the photo and is scored against the actual photo. He and Anna can agree on whatever they like — the photo is still the photo. The best they can do when unsure is a vague sketch. That’s the blur.
Two families. The second is what JEPA actually uses, and it’s a genuinely odd trick.
Alongside every matching pair, feed it mismatched ones — the left of this photo, the right of that one — and demand those be scored far apart.
Now “everything is 7” fails instantly: if all codes are identical, mismatched pairs are close too, and the model is punished. Straightforward, and it works.
The cost: as the code gets richer, the number of wrong examples you need to pin it down explodes. It stops being practical exactly when you want the codes to be detailed.
Don’t use one encoder for both sides. Use two — and make the one that produces the right answer a slow-motion copy of the one that’s learning.
The student sprints; the answer key strolls along behind it, always a few steps back. To collapse, both have to be sitting on the same constant at the same moment — but every time the student moves toward it, the answer key is still back where the student was a moment ago, and the error goes up, not down. The shortcut stops paying.
This is what I-JEPA and V-JEPA use. BYOL and data2vec use it too.
Nobody fully agrees on why the lagging-copy trick works. It reliably does, across many labs and architectures, and there are several competing explanations — but it was found by experiment, not derived from theory. If it feels slightly like a magic trick, that’s not you misunderstanding it. It is a bit of a magic trick.
He saw this hole before anyone built the thing. The position paper doesn’t list one objective, it lists four — and two of them exist purely to stop the model taking the shortcut. The fix isn’t a patch applied afterwards; it’s in the specification.
| # | what it asks for | why it’s there |
|---|---|---|
| 1 | the gist of the visible half must carry real information about it | stops collapse |
| 2 | the gist of the hidden half must carry real information about it | stops collapse |
| 3 | one gist must be predictable from the other | this is the actual learning |
| 4 | keep the randomness dial small | stops a second collapse — see below |
Criterion 3 on its own is the thing that gets gamed. One and two are the guards. Notice they are not “detect cheating and punish it” — they are a positive requirement that the numbers mean something, which cheating happens to fail.
Push mismatched pairs apart. It works. But the number of wrong examples you need grows exponentially with how detailed the gist is — so it fails exactly when you want rich representations. That’s LeCun’s stated objection, not a vague preference.
Don’t generate wrong answers at all. Add a term that directly demands the gists vary and don’t duplicate each other. No curse of dimensionality. VICReg, from his own group, is the worked example; the lagging-copy trick in I-JEPA and V-JEPA is the pragmatic cousin.
Worth knowing, because it catches people who’ve only learned about the first one. The randomness dial — the latent variable — exists so that one input can produce several plausible futures. Give it too much capacity and the model can produce any output from any input.
Then everything is plausible, the score is low everywhere, and you are collapsed again — through a completely different door. You didn’t make all the answers identical this time; you made all the answers acceptable. That is what criterion 4 is standing guard over.
The regularised story is the principled one, and it is the one LeCun argues for. The lagging-copy trick is the one that is actually winning in the benchmarks. Nobody fully agrees on why it works — it was found by experiment and the explanation is still contested. Both facts are true at once, and it is worth holding them together rather than picking the tidier one.
JEPA scores itself by comparing its prediction to its own summary of the answer. There is no ground truth here: the “correct” gist is simply whatever the encoder says, so the model controls both sides of the comparison. The cheapest way to score well is therefore not to predict better — it is to make every summary identical, at which point every answer is right, the score is perfect, and the summaries mean nothing. This is collapse. Not a vague answer; the disappearance of the answer. The fix is to make the “correct” gist a slowly-trailing copy of the model, so standing still never wins.