# Transformers & LLMs

> Build transformer components from scratch and work the full LLM pipeline: tokenization, attention, fine-tuning, alignment, evaluation.

- Source: https://stevenfoerster.com/tutorials/paths/transformers-and-llms/

Learning Path

Build core transformer components from scratch and work through the full LLM pipeline: from tokenization and attention through fine-tuning, alignment, and evaluation.

8 available [Start the path](https://stevenfoerster.com/tutorials/transformers-and-llms-series-introduction/)

1.  - 1
    - beginner

    ## [Transformers & LLMs: Series Introduction and Environment Setup](https://stevenfoerster.com/tutorials/transformers-and-llms-series-introduction/)

    An overview of the Transformers and LLMs series: what it covers, who it is for, how the companion code is structured, and how to set up your environment.

2.  - 2
    - beginner

    ## [Tokenization and Attention from Scratch](https://stevenfoerster.com/tutorials/tokenization-and-attention-from-scratch/)

    Build the four foundational transformer components from scratch in NumPy: tokenization, word embeddings, self-attention, and positional encoding.

3.  - 3
    - intermediate

    ## [BERT Fine-Tuning and Position Embeddings](https://stevenfoerster.com/tutorials/bert-fine-tuning-and-position-embeddings/)

    Explore encoder-only transformers through BERT: inspect masked-LM predictions, fine-tune on sentiment, compare position embeddings, and benchmark distillation.

4.  - 4
    - intermediate

    ## [LLM Decoding and Prompt Strategies](https://stevenfoerster.com/tutorials/llm-decoding-and-prompt-strategies/)

    Compare greedy, beam search, top-k, and nucleus decoding on GPT-2, visualize MoE routing, and test zero-shot, few-shot, and chain-of-thought prompting.

5.  - 5
    - intermediate

    ## [Efficient Fine-Tuning with LoRA and Quantization](https://stevenfoerster.com/tutorials/efficient-fine-tuning-with-lora-and-quantization/)

    Fine-tune a language model with LoRA on a fraction of its parameters, compare FP32/FP16/INT8/NF4 inference, and weigh Flash Attention's tradeoffs.

6.  - 6
    - advanced

    ## [Preference Tuning with DPO](https://stevenfoerster.com/tutorials/preference-tuning-with-dpo/)

    Implement the third stage of LLM training: train a reward model on preference pairs, run DPO to align without reinforcement learning, and compare to best-of-N.

7.  - 7
    - intermediate

    ## [Chain-of-Thought and Reasoning Evaluation](https://stevenfoerster.com/tutorials/chain-of-thought-and-reasoning-evaluation/)

    Compare direct and chain-of-thought prompting on math, implement self-consistency via majority voting, and evaluate code generation with Pass@K.

8.  - 8
    - intermediate

    ## [Building a Tool-Calling Agent with RAG](https://stevenfoerster.com/tutorials/building-a-tool-calling-agent-with-rag/)

    Build a RAG pipeline with ChromaDB, implement a ReAct-style tool-calling agent, and measure retrieval with precision, recall, MRR, and nDCG.


[All tutorials and learning paths](https://stevenfoerster.com/tutorials/)