[Training a 500B LLM from Scratch] 17/25: Who Starts All 5,120 GPUs Together?

The training recipe is saved: 8,192 tokens per sequence, 24 sequences per group, and 80 groups contributing to one update. The file is in front of me; 640 servers are in the data center. One real submission still separates the plan from “start.”

Starting a program on one machine is familiar. Starting across 640 cannot mean opening a terminal on each. They need the same environment, knowledge of their partners, and a clear assignment of who owns these resources now.

I turn “start training” into one complete job: request machines, launch programs, let processes find their partners, and verify that every card completes the same update. On exit, the machines must also be returned cleanly for the next job.

Give One Scheduler Responsibility for the Machines

The first stop is Slurm, selected in Part 14. A request states node count, GPUs per node, and expected duration. Slurm queues it according to resources and priority, then allocates the machines when the conditions are met.

The main run requests 640 B200 servers with eight GPUs each, 5,120 total. It receives one complete allocation. Colleagues use the same entry point, making it possible to see which machines are busy, which are under repair, and when another job might run.

Requesting 640 nodes sounds like a long wait. But this computation already has a TP8, PP8, DP80 layout and needs the full group ready. Every update involves all parallel groups; missing members leave others waiting. Starting 200 nodes and gradually adding the remaining 440 would require an implementation that adjusts groups and training state. This plan uses a fixed scale so batch size, learning rate, and data progress remain explicit within a run.

The equipment list also includes 20 spare servers. I retain their role as failure replacements. The main allocation remains 640. Spares receive health checks and may run acceptance work that can stop immediately, but their 160 GPUs are not promised long-term to a second critical job.

Slurm's resource documentation explains GPU registration, allocation, and isolation. Acceptance must also compare registered GPUs with actual devices. Machines under repair leave the allocatable pool and return after repair and testing.

Scheduler: Software that uses resources, priorities, and task requirements to decide which job runs on which machines.

Job: A complete task submitted to a scheduler, with its own identifier, resource request, and execution records.

Partition: A collection of Slurm nodes and their usage rules. Here, the training partition manages production training machines.

Allocating Machines Does Not Yet Start the Program

The job is registered as 4821. Its status, logs, and usage duration can now be found through that identifier. Once the machines are allocated, the program still needs to start.

Each server contains eight GPUs, and I assign one training process per GPU. That requires 5,120 processes across 640 machines. A process is a running instance of a program. Each runs the same training code, then takes responsibility for a different portion according to its identifier.

Two places in a launch script can create processes. Slurm can start eight tasks per node, or it can start one launcher that creates eight processes. Multiplying by eight in both places launches duplicates. I choose the latter: Slurm starts one torchrun per node, which creates eight local GPU processes. That is 640 launchers and 5,120 training processes.

This assigns local process numbering to torchrun, while the Megatron entry point uses distributed-process initialization. Slurm allocates machines and starts node-level tasks; the launcher manages eight children within each node. A small deployment test verifies visibility: each launcher sees its eight assigned GPUs, and each child uses a different card.

PyTorch's torchrun documentation explains process identifiers and rendezvous between nodes. This plan fixes 640 nodes and eight processes per node, using one shared rendezvous address and port without joining two jobs into the same communication session.

Process: An instance of a program being executed by the operating system. Each GPU process here handles training on one GPU.

Launcher: A program that creates and manages training processes. Here, it is torchrun.

Rank: A process identifier within a distributed task. Global rank identifies its place in the job; local rank identifies its GPU within the node.

How Does Each Process Know Its Work?

I follow a log entry: job 4821, server train-032, local GPU 3, with a global rank alongside it. Program, machine, and card can now be matched.

Next, check its partners. Megatron reads Part 15's parallel configuration and establishes several relationships: eight GPUs cooperate within a layer, eight stages form a pipeline, and 80 model replicas combine gradients for corresponding parameters. The researcher specifies group sizes; the framework creates the communication groups.

Those groups must occupy the right places. A TP group belongs inside one server, using its fast internal links. PP stages span eight servers. DP links corresponding parameters across 80 pipelines. This requires checking framework rank order against actual node order. Seeing “TP=8” alone is insufficient; startup records must locate each member on a machine.

I have startup print a compact mapping: rank, server, GPU, TP group, PP stage, and DP identifier. Select one pipeline and verify it covers exactly eight servers. Select one TP group and verify it stays inside one. Placement mistakes can then be found before computation begins.

Once members are established, NCCL turns these cooperation requirements into GPU transfers. It does not need to know why we are training 500B; it exchanges and combines data as requested. Drivers, topology, and communication settings determine the NICs and paths used. Part 12's network and Part 14's libraries now serve Part 15's specific division of work.

Megatron's parallel-state implementation provides the group construction to check. The production script pins this version and saves the mapping with the configuration.

The drawing expands one of 640 servers. Numbers 0 through 7 are local GPU process identifiers, each corresponding to a card. The launch pattern repeats across all nodes. Megatron then forms TP, PP, and DP groups, and NCCL performs their communication.

Communication group: Processes that cooperate in a particular data exchange. One process can belong to separate TP, PP, and DP groups.

Give Every Machine the Same Tools and Every Run Its Own Records

The placement is correct; the tools must also agree. All 640 machines use the training image built in Part 14. Pyxis connects container startup to Slurm, and Enroot runs the environment. The image digest fixes the content so machines can verify what “the same version” means.

I separate container access into three locations: read-only material, read-only code and configuration, and a writable directory for this run. Material and tokenizer versions stay fixed. Logs and checkpoints go to an independent path such as runs/500b-main-001/. A new comparison experiment gets a new directory so two jobs cannot overwrite the same progress.

Distribute the image before startup. Having 640 machines download from an external registry simultaneously would begin training with a network queue. Verify images and cache data first. Image digest, code commit, model configuration, data manifest, and launch arguments together form the run record.

The Pyxis project supplies the Slurm container integration. It must match the installed Slurm version, so scheduler upgrades include the plugin in the same validation round. Keep Part 14's pinned toolchain, test it in a small partition, then roll it into the main training partition.

Image digest: An identifier derived from image content, used to verify that nodes receive the same artifact.

Mount: Making a host or shared-storage path accessible inside a container.

What Goes into the Submission Record?

Once resources, processes, and environment agree, I collect them into a run sheet. The numbers from earlier articles fill its fields; administrators supply site partition names, paths, and network settings.

The time limit introduces a number different from the training calendar. Main training takes about 76 days, but a site's maximum job duration may be shorter. I divide execution into allowed segments connected by complete checkpoints. Requested duration, advance termination signals, and time reserved for saving go into the site configuration, followed by a real stop-and-resume drill.

Slurm's sbatch documentation defines node, task, time, and signal options. Review submission and training arguments together so requested resources and the number of processes launched agree before execution.

Where Does Other Work Run During Main Training?

Allocating all 640 servers immediately leaves evaluation looking for a place. The GPUs are occupied, and evaluation time needs a position in the plan too.

I keep data preparation and lightweight checks on CPU services. Large-model evaluation receives scheduled milestone windows after complete checkpoints are saved. Those windows count toward calendar efficiency. Concurrent evaluation, teacher-data generation, or new experiments require a separately requested compute pool with its cost and resources added explicitly.

Slurm queues and backfilling can use short gaps. Main training retains priority; interruptible experiments have defined limits. Any use of spares accepts failure replacement as the higher priority.

The task list now shows where resources go: training, maintenance, evaluation, and the reasons queued work has not started. Waiting becomes visible in the plan.

Prove the Whole Path with a Small Job First

With the run sheet ready, I request only two servers through the same submission path. The image, directories, and launch method stay the same. Confirm that 16 processes each occupy one GPU, logs are complete, and deliberate cancellation exits cleanly and releases resources. Then expand to an acceptance scale suitable for the target model to test actual groups, saving, and recovery.

Starting every process is not the finish. Wait for the same update to complete: loss is logged, input-token counts advance, update counts agree, and no machine remains stuck in initialization. That moment establishes that the machines have begun training together.

The path from one configuration to 5,120 GPU processes is connected: Slurm assigns machines, Pyxis and Enroot supply the environment, torchrun launches processes, Megatron assigns computation, and NCCL exchanges results.

Logs begin to scroll, bringing the next question. Does being able to start justify committing the entire cluster for 76 days? Before investing that time, we need to accept the system one scale at a time.