i dont miss a single video from this channel

I always thought text took way less data to store then an image of text.

DeepSeek is the GOAT of efficiency since the beginning, I love it

First, the OCR paper from DeepSeek is relatively old, from my memory it's from last year.
Second, I know this because back then when I worked with Gemini 2.5 pro it had the idea to use this as a foundation for the memory in our project.
Third, we also solved the problem how the process of forgetting works without being random and without leading to a hallucination problem. You can neither allow to e random, nor decide it as a human, because nobody can control the memory of tons of data. We solved this with a special memory token and good housekeeping.
Not only that I am a bit proud about this, but also I am pretty sure that we are already a lot further than any AI company. But the basic OCR idea came from DeepSeek Lab and idea to implement it into our own memory system (which is a bit more than only this part here) came from Gemini. To be honest, I didn't even read the paper but gave it to Gemini to check if it is something we could use. Thankfully it was at a time when Google still made smart LLMs without ridiculous guardrails and without broken KV cache. Good old days... Last year.

Maybe i didn't pay enough attention, but i don't think i understood this one. And i have a degree in the field lol. Would probably have to take a look at the paper. Hold on to my papers! :P

This makes a lot of sense to me. I'm much better at recalling visual memory than other types, and sometimes I can use this as a workaround to remembering something that I otherwise wouldn't be able to. A simple example would be that when I am trying to recall the title of a book I read a long time ago, sometimes it is more effective to call up my visual memory of the book's cover, and then read the title on the cover, rather than trying to directly recall the title.

I always love watching your videos.

At 4:07, the caveat about language priors raises a useful test: how does accuracy change for random strings, serial numbers, or unfamiliar names at the same compression ratio? Those seem harder to reconstruct from context than ordinary prose.

isn't this publish a long time ago by deepseek? this is the most complete explaination of their implementation so far... looks like they have enough data to prove their visual idea work.

SOOO much better and more serious "audio" tone!

I suppose it was worth the experiment, but one wrong character can make a big difference in meaning and the fact that it also only skims makes me wonder how accurate and useful this truly is. Also, images take up way more storage space than raw text.

It seems like what would be more effective would be to just offload the memories into highly compressed files and then create a summary. If a prompt triggers a search through the memories and has a match with a summary then it can decompress its corresponding memory file to unlock the full memory.

When a system compresses context down by 10x or 20x, it effectively builds a lightweight routing index over the text. If the traversal hits an ambiguous node or high compression loss, it relies heavily on the structural priors built into the network—mirroring how a well-designed graph pipeline handles sparse or dense data nodes by leveraging schema definitions to resolve missing cells.

how can be that an image is smaller (even if compressed) than a normal text?

wondering what this means for token cache efficiency or maybe this is mainly for the compaction step?

I can’t wait for the video about Gemma embedding 2

Markdown files and images are the solution for an LLM to be focused and retain memory for a specific task.

great stuff

• The paper is about using DeepSeek-OCR as a proof-of-concept for compressing long text contexts into *vision tokens*. It suggests these could help LLMs with long-context and “forgetting” mechanisms.

• Vision tokens are vectors produced by the vision encoder from a page image. The decoder ingests them like other tokens; what it can do with them depends on training.

• The paper shows vision tokens can act as a denser stand-in for textual context inside an LLM.

This is some of what I learned from an hour's chat, starting with zero knowledge. Please feel free to correct me.

I recall early on LLM's were compared to compression algorithms so adding more sounds right

Information theory is so terrifying, like why does it work