Module 03: Transformers and LLMs

Goal

Understand transformer architecture, LLM behavior, prompt design, structured outputs, and cost-aware inference.

Core Topics

• Tokenization.
• Embeddings.
• Positional encoding.
• Self-attention and multi-head attention.
• Feed-forward networks.
• Residual connections and layer normalization.
• Encoder vs decoder models.
• Causal language modeling.
• Pretraining, instruction tuning, fine-tuning, LoRA, QLoRA.
• Quantization and inference optimization.
• Context windows, temperature, top-k, top-p.
• Hallucination and evaluation.

Study Tasks

• Implement scaled dot-product attention.
• Implement a tiny transformer block.
• Call at least one hosted LLM API.
• Run at least one local or open model if hardware allows.
• Create structured JSON output prompts.
• Build a function-calling example.
• Compare latency, cost, and quality across model choices.

Prompt Practice

• Write prompts with clear role, task, context, constraints, and output format.
• Use few-shot examples only when they improve reliability.
• Test prompts against failure cases.
• Version important prompts like code.

Project

Build projects/03-transformer-llm-lab/.

Tips

• Learn attention from scratch before depending on frameworks.
• Separate prompt quality from model quality during testing.
• Always validate structured outputs with schemas.
• Track token usage early; cost surprises are architecture bugs.

Recommended Resources

• The Illustrated Transformer.
• Attention Is All You Need.
• Hugging Face Transformers documentation.
• Provider documentation for structured outputs and tool calling.

Completion Checklist

• Can explain self-attention.
• Can explain why context windows matter.
• Can build structured LLM outputs.
• Can compare model trade-offs.
• Can identify hallucination risks.