[Training a 500B LLM from Scratch] 13/25: How Do We Keep Data Flowing and Save Training Progress?

The machines have places in the facility and routes between servers. I spread out the acceptance procedure for the work before training, locating where material enters the GPUs and where parameters are saved. The 5,120 B200s need to train for about 76 days. Both paths must work before the run begins.

The original acceptance sheet had only two entries: “read succeeded” and “save succeeded.” Both sounded right, yet neither answered the important questions. If the next batch arrives slowly, how long do the GPUs wait? The calendar extends to 76 days; if training stops on day ten, where can it resume? I add timing and recovery requirements, and the storage task becomes concrete.

Those questions divide storage into two jobs: keep material arriving and preserve what the model has learned. Follow the read path first and the save path second. Space, speed, and equipment can then be calculated from the work they must perform.

Where Can the Next Batch Get Held Up?

Start with the material about to enter the GPUs. Part 9 converted text to token IDs and organized training sequences. A file's main content resembles [418, 732, 25, …], a sequence of integers whose actual values depend on the tokenizer.

Complete files reside in shared storage, with all 640 training servers drawing from the same version of the library. The training program assigns material according to the recipe. With one authoritative source established, the next question is how each batch arrives on time.

I divide the route into stages: fetch upcoming files to the server's own SSD, load the current batch of IDs into system memory, then transfer it to the GPU. While one batch computes, the next is prepared. Reading and computation can now fit together.

Returning to shared storage for every read makes too many trips. Material is shuffled, mixed, and sometimes reread, so upcoming files belong closer to the server. Each machine gets an initial 1TB local cache, filled and replaced according to the sampling plan.

The full 5T-token inventory stays in shared storage. Servers fetch files as needed; GPUs receive only the current small batch. The relay runs from shared library to local drive, memory, and GPU. We can now observe and improve the preparation time for the next batch.

Shared storage: A file storage system that multiple servers can access. Here, it holds the common training material and recent training progress.

Cache: Space used to keep upcoming content closer to where it will be used. Here, it is on each server's own solid-state drive.

Prefetching: Preparing the next batch of material before processing of the current batch finishes.

Why do 5T tokens take up only 20TB?

Back on the capacity sheet, 5T and 10T appear together. Part 7 set a target of about 5T distinct qualified tokens, while training reads 10T cumulatively, revisiting some material.

Reading a five-million-page collection for ten million page reads does not require a second collection on the shelf. The sampler finds the original page again. Disk capacity follows distinct inventory; training volume counts every token read.

Now include the ID format. Each takes four bytes in this plan, so the 5T-token body occupies:

5 trillion IDs × 4 bytes each = 20 trillion bytes = 20TB

Twenty terabytes is smaller than intuition suggests. Look inside the file again: it holds IDs. The model looks up their vectors during training. Each token occurrence need not carry a complete vector inside the dataset, which explains the size difference.

Beyond the IDs, the program needs indexes to find document positions and lengths quickly. I reserve 30TB for the official material: 20TB for IDs and the remainder for indexes, document boundaries, and supporting files.

Raw pages and versions before and after cleaning go into separate archives. The active workspace holds the selected version, so the GPUs receive the dataset chosen for this run.

Why use four bytes per ID?

The ID format must match the vocabulary. Megatron's dataset implementation selects integer types by vocabulary size. This plan uses four-byte int32 for a large vocabulary. Once files are built, check tokenizer, type, and actual file size together, grounding the capacity calculation in the delivered files.

TB: We use decimal units throughout this article. 1TB equals 1,000GB. A figure such as 20TB describes how much space a file takes up.

Index: Information about document locations and lengths that helps a program find the content it needs directly.

How do we save what the model has learned by day ten?

With the read path established, turn the training calendar to day ten. The parameters will have changed many times by then; retaining only the initial model cannot preserve that work. “Save succeeded” must therefore include a complete record of where training has reached.

That record is a checkpoint. It retains parameters, training step, and data-reading position so recovery knows where to continue. Begin by counting the most visible component: the weights.

A 500B model has 500 billion parameters. Save each in two bytes:

500 billion × 2 bytes = 1TB

Saving 1TB of weights still leaves training incomplete. The optimizer uses past updates to decide its next adjustment. Without those records, its original update process cannot resume.

Open one parameter's ledger to see what is missing. Computation uses a compact value; updating retains a more precise copy to preserve small changes. Adam also keeps two histories, tracking the direction and scale of gradients to guide direction and step size. Part 16 follows one value through an update. For now, copy that ledger across 500 billion parameters to obtain the storage requirement.

With the common mixed-precision Adam setup selected, save three main components at the end of a complete training step:

The total is 7TB. Reading required only a small batch of material; saving progress preserves the current training state of the entire 500B model. The write suddenly becomes much larger.

The cluster shares this 7TB, each process writing its own shards. Together they form a complete checkpoint, accompanied by step count, learning-rate schedule, random state, and data position. NVIDIA's checkpoint documentation includes these in complete recovery so the next start has a defined place to resume.

The earlier training-memory calculation used 8TB. The missing terabyte has a specific reason: gradients remain in memory to perform an update, but this checkpoint is written after that update. The next step calculates new gradients. Main saved state is therefore 7TB, with runtime memory and checkpoints counted at their respective moments.

Checkpoint: A saved copy of the model and training state at a particular point. Restoring it allows training to continue from that point.

Optimizer: An algorithm that adjusts model parameters using calculated gradients. Its historical state affects subsequent adjustments.

BF16 and FP32: Two numerical precision formats. Here, they use two and four bytes per number, respectively.

Why Do Three Retained Checkpoints Need Space for Four?

Next comes the save interval. I begin with one hour, retaining three recent copies in fast shared storage. If 10:00 completes and a machine fails at 10:40, recovery returns to 10:00 and repeats 40 minutes. That establishes a much smaller recovery boundary than restarting ten days of work.

Shorter intervals generally reduce recomputation after failure and increase writes. One hour is a starting point. Trial save and recovery times, together with observed failures, will determine a better interval from actual costs.

Writing “three copies” overlooks the fourth one being created. All three old copies stay until the new one finishes and passes validation. Only then can the oldest go. Simultaneous space is therefore:

3 completed checkpoints + 1 being written = 4 × 7TB = 28TB

Beyond 28TB, allow for supporting files and the writing process. Recent checkpoints receive 35TB.

But three copies on one shared system still leave only one recovery route if that system fails. Each day, select a complete checkpoint and copy it to an independent archive, giving long-term progress another home.

For 76 training days, one copy per day, with 7.5TB per copy including supporting files, the archive calculation is:

76 checkpoints × 7.5TB = 570TB

Round up to 600TB, or 0.6PB, dedicated to those daily checkpoints. Raw material and cleaned versions have a separate archive allocation.

Recent copies support quick restarts. Daily copies preserve longer-term progress and recovery from shared-storage failure. Each location now has a specific purpose.

Archive: Storage for files that we need to retain over time but read less frequently. The checkpoint archive in this plan is on an independent system.

PB: 1PB equals 1,000TB. Here, 0.6PB means 600TB.

We have enough space. How long will we wait?

The space is accounted for, but the acceptance sheet's time column is empty. A library can fit yet load too slowly at startup; a checkpoint can fit yet hold up training. Each direction needs a time target.

First calculate sustained input: 10T IDs at four bytes each, about 40TB over 76 days, averaging roughly 6.1MB/s. That modest average reflects the large amount of model computation performed on each batch of IDs.

At a cold start, however, all 640 caches are empty. The average cannot explain that queue. Set a separate concentrated-loading test: read a 20TB ID body using the actual distribution pattern, targeting ten minutes:

20,000GB ÷ 600 seconds ≈ 33.3GB/s

Round the acceptance target to 40GB/s. It covers concentrated startup loading; 6.1MB/s covers average ID delivery across 76 days. They operate on different time scales, which explains the gap.

Real distribution also reads indexes and may send one file to several servers. Count duplicate transfers and record when each machine becomes ready. That reveals the actual startup wait.

For the hourly 7TB checkpoint, allow ten minutes for the entire save, including state preparation, transfer, checks, and final confirmation. I allocate five minutes to the transfer itself:

7,000GB ÷ 300 seconds ≈ 23.3GB/s

This gives a rounded sustained-write target of 25GB/s. Keep training reads running while the checkpoint writes and watch whether GPUs wait. Overlap background saving where possible, while recording the actual training pause separately.

The purchase request now describes specific work: load material on time and make a roughly 7TB checkpoint complete and usable within ten minutes. The 40GB/s read and 25GB/s write targets serve those tasks, tested with our files and program.

GB/s: Gigabytes transferred per second, a measure of read or write speed. TB tells us how much fits; GB/s tells us how fast it moves.

Asynchronous saving: First capturing a consistent training state, then continuing file writes in the background so that part of the save can overlap with training.

Which Equipment Meets These Requirements?

With capacity and speed defined, equipment selection has a basis. Material and recent checkpoints require 30TB plus 35TB, or 65TB, with working space for conversion and version changes. I place DDN AI400X3M's 120TB usable configuration in the procurement plan.

DDN's official 2026 introduction lists 120TB, 250TB, and 500TB usable options. Choose 120TB first and assign each portion to present work:

Each server's 1TB cache and the independent 0.6PB archive also receive their places on the facility plan. The 640 caches total 640TB, spread across machines and potentially containing duplicate files. Shared workspace remains a separate 120TB allocation.

Follow blue from shared storage through local cache and memory into the GPU. Follow orange back as training state reaches shared storage and a daily copy goes to the independent archive. The three locations form a complete return path around one training run.

Follow blue lines for training data and orange lines for saved progress. The equipment layout illustrates the plan. Click to view the full-resolution image.

Delivery must provide a complete system accessible to all 640 servers. This 120TB configuration undergoes concurrent 40GB/s reads and 25GB/s writes, with software, connections, and endurance included in integration requirements.

Include these items in the purchasing and integration checklist

To make the directories accessible across machines, the order includes AI400X3M, ES400NVX2 management nodes, EXAScaler filesystem software, and support. After system and data-protection overhead, files must have 120TB available.

Files also need their own network. Part 12's ports are assigned to GPU communication. Storage requires independent 200GbE with additional NICs and switches, verified against remaining slots and PCIe paths. One 200Gb/s link provides about 25GB/s nominally and less in file throughput. Multiple storage-side connections must support cluster concurrency, followed by end-to-end acceptance.

The local 1TB initially comes from the SSDs included in each server. System usage, drive count, and usable space enter the integration list. A replaced server can refill its cache from shared storage, which remains the authoritative dataset source.

Capacity is not enough; drives must endure repeated writes. At 7TB each hour, logical writes reach about 168TB per day. Deleting old copies frees space but does not undo wear. The supplier checks endurance and replacement arrangements against actual size, frequency, and service duration.

Platform peaks help shortlist candidates. Final acceptance still makes this 120TB configuration read material and write real checkpoints together, recording sustained throughput, GPU waiting, and complete save time.

Usable capacity: Space that can actually be allocated to files after system and data-protection overhead.

Finally, stop a run and resume it

Writing files leaves one crucial question: can they actually restart training? I put a full drill into acceptance. Run, save, deliberately stop, replace one position with a spare, then recover from the checkpoint.

Mark a checkpoint recoverable only after all shards exist and checks pass. After restart, compare parameters, optimizer state, steps, and data position. Continue a short distance and compare with an uninterrupted control, using the implementation's accepted numerical tolerance.

Repeat from shared storage and the independent archive so both recovery paths are exercised. Files in a directory are an intermediate result. Continuing training completes the handover.

The arrangement now consists of 120TB shared storage, 1TB per-server cache, and a 0.6PB independent daily archive. Material arrives in batches, progress leaves each hour, and a stopped machine has a route back. Hardware and data now have their paths. One layer remains: how will the software on all these machines execute the same training job?