Learning Path

Transformers & LLMs

Build core transformer components from scratch and work through the full LLM pipeline: from tokenization and attention through fine-tuning, alignment, and evaluation.

8 available Start the path
  1. 1 beginner

    Transformers & LLMs: Series Introduction and Environment Setup

    An overview of the Transformers and LLMs series: what it covers, who it is for, how the companion code is structured, and how to set up your environment.

  2. 2 beginner

    Tokenization and Attention from Scratch

    Build the four foundational transformer components from scratch in NumPy: tokenization, word embeddings, self-attention, and positional encoding.

  3. 3 intermediate

    BERT Fine-Tuning and Position Embeddings

    Explore encoder-only transformers through BERT: inspect masked-LM predictions, fine-tune on sentiment, compare position embeddings, and benchmark distillation.

  4. 4 intermediate

    LLM Decoding and Prompt Strategies

    Compare greedy, beam search, top-k, and nucleus decoding on GPT-2, visualize MoE routing, and test zero-shot, few-shot, and chain-of-thought prompting.

  5. 5 intermediate

    Efficient Fine-Tuning with LoRA and Quantization

    Fine-tune a language model with LoRA on a fraction of its parameters, compare FP32/FP16/INT8/NF4 inference, and weigh Flash Attention's tradeoffs.

  6. 6 advanced

    Preference Tuning with DPO

    Implement the third stage of LLM training: train a reward model on preference pairs, run DPO to align without reinforcement learning, and compare to best-of-N.

  7. 7 intermediate

    Chain-of-Thought and Reasoning Evaluation

    Compare direct and chain-of-thought prompting on math, implement self-consistency via majority voting, and evaluate code generation with Pass@K.

  8. 8 intermediate

    Building a Tool-Calling Agent with RAG

    Build a RAG pipeline with ChromaDB, implement a ReAct-style tool-calling agent, and measure retrieval with precision, recall, MRR, and nDCG.

All tutorials and learning paths