Huiyu Chen

Meta · Multimodal LLMs

← All learning paths阅读中文版 →

PATH 02 · 8 CHAPTERS

LLM infrastructure, from memory to throughput

vLLM, SGLang, TensorRT-LLM, Megatron-LM and DeepSpeed, with GPU/TPU architecture, kernels, sharding and profiling.

  1. 01

    Why serving is slow: prefill, decode, KV cache and queues

    Build a latency and memory budget before discussing continuous batching, paging and prefill/decode separation.

  2. 02

    vLLM: from PagedAttention to measurable serving

    Connect the scheduler, KV block manager, model runner and API server, then design a fair throughput experiment.

  3. 03

    SGLang: shared prefixes, structured generation and PD separation

    Understand RadixAttention through repeated prefixes, then budget the cost of transferring KV between workers.

  4. 04

    TensorRT-LLM: hardware-aware optimization and quantization

    Understand fusion, quantization, kernel selection and runtime scheduling, then compare gains at matched quality.

  5. 05

    Megatron-LM: what do TP, PP, DP and sequence sharding split?

    Split one matrix multiplication, budget communication and understand pipeline bubbles and context parallelism.

  6. 06

    DeepSpeed ZeRO: account for every training state

    Start with Adam weights, gradients and optimizer state; derive ZeRO-1/2/3 memory savings and communication costs.

  7. 07

    GPU, TPU and kernels: why FLOPs do not explain speed

    Use Roofline to understand HBM, on-chip storage, matrix units, tiling and fusion, then locate Triton and Pallas in the stack.

  8. 08

    XProf / Nsight: find the bottleneck in a timeline

    Inspect end-to-end timelines before slow kernels; use warm-up, synchronization and controlled experiments to avoid timing traps.