Train a 500B LLM from Scratch with Me (25 Chapters)
- Which GPUs Should We Buy?
- How Many GPUs Do We Need?
- What Is the Full Budget?
- What Is Inside a Large Language Model?
- How Does MoE Compute with Only Some of Its Parameters?
- How Is Attention Trained?
- Where Does the Training Data Come From?
- How Do We Clean the Training Data?
- How Do We Split Long Documents and Build a Training Mix?
- How Can Existing Models Help Us Write Training Material?
- How Do We Prepare a Data Center for These GPUs?
- How Do We Install the NICs and Cables for 5,120 GPUs?
- How Do We Keep Data Flowing and Save Training Progress?
- How Do We Install the Software on Our First Server?
- How Do We Split One Model Across 5,120 GPUs?
- How Much Do We Read, and How Do We Update the Model?
- Who Starts All 5,120 GPUs Together?
- When Can the Whole Cluster Begin Training?
- Training Stopped. How Do We Pick Up Where We Left Off?
- The GPUs Look Busy. Why Is Training Still Slow?
- After Reading So Much, How Does the Model Learn to Answer Questions?
- How Can a Model Learn from the Results of Its Own Attempts?
- How Do We Know the Model Is Ready?
- How Do We Put the Model in Users' Hands?
- Once the Model Exists, How Does the Investment Keep Creating Value?
New ideas, straight to your inbox.
AI, engineering, and experiments. One or two useful emails a month.
No spam. Unsubscribe anytime.