Huiyu Chen

Meta · Multimodal LLMs

← All learning paths阅读中文版 →

PATH 03 · 5 CHAPTERS

Multimodal understanding, from pixels to meaning

Image patches, CLIP, masked reconstruction, vision-language models and video timelines: understand what a model sees.

  1. 01

    How images become tokens: patches, positions and resolution

    Trace RGB tensors into ViT patch embeddings, count visual tokens and inspect information lost through resizing and cropping.

  2. 02

    CLIP: bring images and text into a shared space

    Construct a B×B similarity matrix and understand contrastive training, negatives, temperature and zero-shot classification.

  3. 03

    Reconstruction: what does a model learn from hidden patches?

    Distinguish pixel reconstruction from semantic prediction; inspect MAE visible-only encoding, lightweight decoding and masked loss.

  4. 04

    Vision-language models: from visual features to answers

    Understand the vision encoder/projector/decoder interface and distinguish alignment, instruction tuning and grounding.

  5. 05

    Video understanding: sampling, temporal grounding, audio and evaluation

    Treat video as temporal evidence, budget frames and tokens, and diagnose static shortcuts, subtitle leakage and missed events.