AI Is Learning at the Wrong Level of Abstraction — Matthieu Wyart
Machine Learning Street Talk
19,390 views
This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlst
Why can deep networks discover abstractions that shallow models miss? Statistical physicist Matthieu Wyart joins Tim Scarfe to argue that the answer lies in the hidden hierarchy of data. Language and images are built from parts within parts; depth lets a network recover those coarse-grained variables and escape the curse of dimensionality.
The conversation moves from jamming transitions and rough loss surfaces to Chomsky, context-free grammars and machine creativity. Wyart explains why next-token prediction can still recover compositional structure, where current systems fall short of genuine scientific invention, and why predicting latent representations rather than raw tokens could make learning far more sample-efficient.
They also examine diffusion models, neural scaling laws and the limits of physics-inspired theory. The final question is on a personal note: if mistakes are the price of leaving the beaten path, how much scientific risk is worth taking?
---
TIMESTAMPS:
00:00:00 Can machines learn abstractions from data?
00:02:00 Notion agentic workspace
00:02:49 From statistical physics to machine learning
00:06:40 What physics can explain about learning
00:16:37 From Carnot to Chomsky bulldozer
00:21:21 How deep networks recover hidden hierarchies
00:32:43 Where machine creativity still falls short
00:40:48 How deep nets escape the curse of dimensionality
00:52:19 Why predict latents instead of tokens
01:02:49 The sample-efficiency case for latent prediction
01:08:31 Diffusion, scaling laws and text entropy
01:16:40 The scientists we learn from and the mistakes we make
---
REFERENCES:
person:
[00:00:43] Noam Chomsky
https://linguistics.mit.edu/user/chom...
tool:
[00:02:08] Notion Developer Platform
https://www.notion.com/en-gb/blog/int...
paper:
[00:04:43] Mastering the game of Go with deep neural networks and tree search
https://www.nature.com/articles/natur...
[00:05:52] Reconciling modern machine-learning practice and the bias-variance trade-off
https://arxiv.org/abs/1812.11118
[00:25:54] How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
https://arxiv.org/abs/2307.02129
[00:42:12] Efficient Estimation of Word Representations in Vector Space
https://arxiv.org/abs/1301.3781
[00:52:46] Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
https://arxiv.org/abs/2301.08243
[00:52:54] Learn from your own latents and not from tokens: A sample-complexity theory
https://arxiv.org/abs/2605.27734
[01:08:31] A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data
https://arxiv.org/abs/2402.16991
[01:11:39] Scaling Laws for Neural Language Models
https://arxiv.org/abs/2001.08361
[01:12:17] Deriving Neural Scaling Laws from the statistics of natural language
https://arxiv.org/abs/2602.07488
[01:13:34] Prediction and Entropy of Printed English
https://ieeexplore.ieee.org/document/...
---
LINKS:
Download PDF transcript: https://app.rescript.info/share/f7644...
20 Comments
For those wondering or placing bets of billions that these LLMs...will one day discover things by themselves, well dream on. They will remain very useful tools that help people with the skills in those fields to do that. e.g. These tools will help trained doctors/researchers...etc in unlocking some discoveries. They will never be able to do it by themselves.
Also, the scaling laws naturally work as more data & compute is added into the equation but it soon plateaus once all patterns have been picked up. It's akin to drinking orange juice, one gets vitamin C up to a point and drinking more is pointless after saturation.
AI Is Learning at the Wrong Level of Abstraction — Matthieu Wyart
Machine Learning Street Talk
19,390 views
This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlst
Why can deep networks discover abstractions that shallow models miss? Statistical physicist Matthieu Wyart joins Tim Scarfe to argue that the answer lies in the hidden hierarchy of data. Language and images are built from parts within parts; depth lets a network recover those coarse-grained variables and escape the curse of dimensionality.
The conversation moves from jamming transitions and rough loss surfaces to Chomsky, context-free grammars and machine creativity. Wyart explains why next-token prediction can still recover compositional structure, where current systems fall short of genuine scientific invention, and why predicting latent representations rather than raw tokens could make learning far more sample-efficient.
They also examine diffusion models, neural scaling laws and the limits of physics-inspired theory. The final question is on a personal note: if mistakes are the price of leaving the beaten path, how much scientific risk is worth taking?
---
TIMESTAMPS:
00:00:00 Can machines learn abstractions from data?
00:02:00 Notion agentic workspace
00:02:49 From statistical physics to machine learning
00:06:40 What physics can explain about learning
00:16:37 From Carnot to Chomsky bulldozer
00:21:21 How deep networks recover hidden hierarchies
00:32:43 Where machine creativity still falls short
00:40:48 How deep nets escape the curse of dimensionality
00:52:19 Why predict latents instead of tokens
01:02:49 The sample-efficiency case for latent prediction
01:08:31 Diffusion, scaling laws and text entropy
01:16:40 The scientists we learn from and the mistakes we make
---
REFERENCES:
person:
[00:00:43] Noam Chomsky
https://linguistics.mit.edu/user/chom...
tool:
[00:02:08] Notion Developer Platform
https://www.notion.com/en-gb/blog/int...
paper:
[00:04:43] Mastering the game of Go with deep neural networks and tree search
https://www.nature.com/articles/natur...
[00:05:52] Reconciling modern machine-learning practice and the bias-variance trade-off
https://arxiv.org/abs/1812.11118
[00:25:54] How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
https://arxiv.org/abs/2307.02129
[00:42:12] Efficient Estimation of Word Representations in Vector Space
https://arxiv.org/abs/1301.3781
[00:52:46] Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
https://arxiv.org/abs/2301.08243
[00:52:54] Learn from your own latents and not from tokens: A sample-complexity theory
https://arxiv.org/abs/2605.27734
[01:08:31] A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data
https://arxiv.org/abs/2402.16991
[01:11:39] Scaling Laws for Neural Language Models
https://arxiv.org/abs/2001.08361
[01:12:17] Deriving Neural Scaling Laws from the statistics of natural language
https://arxiv.org/abs/2602.07488
[01:13:34] Prediction and Entropy of Printed English
https://ieeexplore.ieee.org/document/...
---
LINKS:
Download PDF transcript: https://app.rescript.info/share/f7644...