[Training a 500B LLM from Scratch] 9/25: How Do We Split Long Documents and Build a Training Mix?

The sorting tutorial had finally passed cleaning. I read through it once more: the first chapter explained the problem, the second developed the algorithm, the third supplied the code, and the fourth tested it. Conditions, reasoning, and answers were all there. It was a complete lesson.

Then the training program hit another obstacle: the article was longer than the current window.

The easiest solution was to fill a window and cut. Before doing that, I wanted to see what kind of article the model would actually receive afterward.

After tokenization, the tutorial contains 12,000 tokens. Its four chapters occupy 3,000, 3,500, 3,000, and 2,500 tokens. Tokens are the small pieces into which text is split, and the unit the training program uses to measure length.

We have room for 4,000 body tokens per window here. Cutting every 4,000 gives us three segments:

1. Segment one: tokens 1 through 4,000. It contains the complete problem statement and the beginning of the algorithm's derivation.
2. Segment two: tokens 4,001 through 8,000. It contains the rest of the derivation and the first half of the code.
3. Segment three: tokens 8,001 through 12,000. It contains the remaining code and all the tests.

The order of the text survived, and all three windows were full. But when I opened the third segment on its own, it began halfway through the code. The problem statement and derivation were in the other two windows.

Those earlier segments do not automatically appear in front of the model when it receives this one. Where we cut, and how much earlier text we carry forward, now affect what the lesson can teach.

Where Should We Cut a Long Article?

Could we simply lengthen the window and preserve the whole article? Sequence length determines how much the model reads at once, and changes the memory and computation required. Longer sequences generally need more intermediate results stored and processed. With full attention, the number of position pairs grows with the square of length. We therefore choose a window suited to the current training stage and put exceptionally long material into the long-context plan.

This cutting plan reserves 4,000 tokens for the body; necessary markers still need room. For the main run, window length is selected alongside the model and memory tests.

Once the length is chosen, we need one ruler. A thousand Chinese characters, a thousand English words, and a thousand lines of code produce different token lengths. Tokenization tells us which sentence will fit.

The most direct approach is the fixed-length cutting shown in the figure. It preserves the article's order and makes a continuous data supply straightforward. It can also cut through a sentence, a derivation, or a code block. Large-scale pretraining can use this approach. NVIDIA's GPT dataset implementation samples by document ID and token offset to assemble inputs of a specified length. To recognize paragraphs or chapters, the program needs additional data-processing rules from us.

For this well-structured tutorial, I moved the cuts to section endings. The four chapters produced bodies of 3,000, 3,500, 3,000, and 2,500 tokens, each fitting within 4,000 body positions. The boundaries between the problem, derivation, code, and tests survived. We had stopped breaking sentences.

Unbroken sentences did not guarantee a complete lesson. The test chapter said “call the function from the previous chapter,” but the function was elsewhere. I brought over that function and the required input conventions, then counted again. When the question and answer fit together, they share a window. When tightly connected chapters exceed the window, the group gets a longer training sequence so the model can see their relationship in the same computation.

The original four chapters still contain 12,000 tokens. Added questions, functions, and explanations increase the processing workload. Each addition gets another fit check; the remaining spaces can later be organized with short material.

The next document was less convenient: one section alone contained 6,000 tokens. I looked deeper for paragraph and code-block boundaries. A unit that must remain continuous but is still too long belongs in the long-context pool. Ordinary continuous text can be cut by token, carrying a small overlap into the next window.

For example, keep the 4,000-body-token limit: the first window reads tokens 1 through 4,000, and the second reads tokens 3,501 through 6,000. The second window begins with the last 500 tokens of the previous segment, then continues with 2,000 new tokens. It now has context near the cut, while its view remains limited to its own window.

4,000 + 2,500 = 6,500 cumulative body tokens

The unique material contains 6,000 tokens. Overlap adds 500 tokens to the processing workload, about 8.3%. Markers and padding are counted separately. We can also use those 500 tokens only as context and turn off their prediction loss. They still consume computation and must remain in the ledger. The 500-token overlap is a choice for this teaching example; whether we need overlap, and how much, depends on the material's dependencies and validation results.

Whether those extra 500 tokens earn their cost depends on the relationship between windows. Parameters accumulate experience through training, but the specific words in one window do not automatically remain in the next window's attention. If earlier text is needed, include it in the current input or use a longer window. We are preparing training samples here. KV cache, which preserves computed context during inference, serves a different stage.

With the cuts settled, I recorded each slice's document ID, starting and ending token positions, and overlap. We can trace every slice to its source and see which content was processed again. Long documents now fit. Short material presents the opposite problem: space left empty. Several short pieces can share a training sequence.

Training window: The content the model can read in one sequence computation. The 4,000 in this article is the example's body-token limit.

Context: The preceding text available to the model at the current prediction position.

Overlap: A passage from the original document included in adjacent training windows to give the later window preceding context.

Why Can the Same Text Have Different Token Counts?

The slice ledger was full of tokens, so I checked the ruler itself. What happens to the same text under a different tokenizer? We run three fragments from the tutorial through the public GPT-2 tokenizer and the public Qwen3 tokenizer. Both tools are public, so the results can be compared directly. The following results were obtained locally on October 19, 2026, without chat templates or special markers.

Both tokenizers split the first sentence into Keep, duplicate, values, and .. The spaces at the start of the middle two pieces belong to those tokens. For this sentence, the counts match. The code shows where the rulers differ: GPT-2 splits sorted(xs) into s, orted, (, xs, and ), while Qwen3 produces sorted, (xs, and ). These pieces can cross the word or symbol boundaries we are used to. Their job is to represent the original text using pieces already in the vocabulary.

The last row made me pause: Qwen3 produced more tokens. Choosing a vocabulary therefore requires comparing Chinese, English, code, numbers, and formulas together to see whether the whole collection grows or shrinks.

Hugging Face's description of the tokenization pipeline separates text processing, splitting, mapping to vocabulary IDs, and adding special markers. In our sorting tutorial, spaces, indentation, and symbols matter. We must be able to recover the original text after processing. All six encodings in the table passed a complete decoding round-trip check. Some tokens contain only part of a character's bytes, so an individual token may look incomplete when displayed alone. The full sequence can still reconstruct the text.

Token: A piece produced when a tokenizer processes text. It may be a whole word, part of a word, a symbol, or a sequence of bytes.

Vocabulary: The list of pieces and their IDs used by the tokenizer.

What Happens When the IDs Enter the Model?

I followed sorted(xs) one step further. In this Qwen3 example, its three pieces have IDs 28298, 53322, and 8. An ID is used for lookup. A larger ID indicates a different position in the vocabulary.

The model receives an ID and retrieves a row of numbers from its input table. That row is the vector we discussed in Part Four. Three tokens therefore become three rows of vectors, which then enter the first attention layer. One token ID is an integer; one token's vector contains many numbers. The figure draws them as small labels and long bars to make these two steps clear.

Once IDs map to vectors, the vocabulary becomes part of the model's organization. When reusing existing weights, we start with that model's tokenizer and special markers. The model has already associated each ID with a row in its input table. If a new tokenizer assigns the same ID to different text, that relationship changes. Extending or replacing the vocabulary requires corresponding input and output layer changes and further training, so it belongs in a separate experiment.

Our from-scratch run can compare vocabularies afresh. I fix a sample covering Chinese, English, code, and mathematics, then compare total token counts, lengths by category, text reconstruction, and vocabulary size. A larger vocabulary needs more space and computation in the input table and output prediction layer. Shorter sequences must earn back that cost. Once selected, the vocabulary files and processing rules are fixed and used to recount the entire inventory.

When integrating an existing model, we can load AutoTokenizer and weights from the same model version and record them as a pair. Our 10T ledger uses this same ruler from the final student model.

Token ID: The integer assigned to a token in the vocabulary, used for lookup.

Embedding: The step that converts a token ID into a vector, and the trainable parameter table used to retrieve those vectors.

Vector: A sequence of numbers. Here, each token retrieves a sequence of numbers representing its features as it enters the model.

How Do Short Documents Fit into One Training Sequence?

With the vocabulary fixed and lengths recounted, the section-based windows still had empty space. Many short documents occupied only a few positions. I placed two short pieces side by side and drew eight boxes to see how they would fit.

Document A contains three body tokens and document B contains two. Each gets an end marker. Separate eight-position sequences require sixteen positions, although only seven contain text or an end marker. Put both documents in one eight-position sequence, and only one position remains empty.

This process is usually called packing. In an implementation with the same fixed sequence length, it reduces the padding positions allocated to these documents. The actual throughput gain must be measured on the chosen training backend. A sequence can contain several documents, each with its own boundary. Standard Megatron pretraining already concatenates documents during sampling. The Megatron Bridge documentation specifically distinguishes this pretraining behavior from the packing switch used during fine-tuning. We need the method supported by our training path.

Less space was wasted, but a new relationship appeared: a sorting tutorial now sat beside a weather report. Should attention read across that boundary?

They were neighbors in position, independent in content. I chose within-document attention first: the weather report reads its own preceding text, and the tutorial reads its own. The figure's two small triangles show that left-to-right reading. The gap between them means no information crosses between documents. An end marker tells the model where a document finishes; the attention mask in the program enforces the reading boundary.

The prediction targets must follow that boundary too. We teach the model to predict the next token within a document, including its end marker. After that marker, we do not use the previous document to predict the first token of the next independent document. That cross-document prediction and padding positions are excluded from loss. Position numbering restarts within each document. NVIDIA's GPT dataset implementation provides position, attention, and loss handling separately; acceptance checks must inspect them separately too.

Related material can use another arrangement. Functions, tests, and documentation from the same code repository need to refer to each other. We first organize them into one ordered training unit with file markers, then allow information to pass within that unit. Isolating independent documents and grouping related files serve different learning tasks.

Training sequence: An ordered sequence of tokens fed into the model at one time.

Packing: Organizing material into training sequences to fill the available positions more fully.

End marker: A special token that marks the end of a document.

Attention mask: The rule that specifies which positions a given position can read.

Loss: A number measuring the difference between the model's prediction and the training target. Training uses it to adjust parameters.

Turn the 10T Ledger into Daily Quotas

The sequences were ready, but the program still faced a choice: should the next sample come from English or code? An improvised order would leave Part Seven's recipe stranded in the spreadsheet.

We keep Part Seven's from-scratch capacity plan, with a workload ceiling of 10,000B tokens. Unique inventory and cumulative reading volume are already separate. Dividing each cumulative allocation by 10,000B gives the initial quotas:

For example, the next 10B training tokens are allocated in that order as 6.16B, 0.79B, 1.40B, 0.50B, 0.99B, 0.15B, and 0.01B. The seven amounts add up to exactly 10B. Code-related pages, source code, and teacher explanations together account for 15%. The 5T unique inventory is a production and acceptance target. If a code-related page occurs in both English and code candidate collections, it belongs to only one pool after deduplication, retaining other attributes as tags.

I kept tokens as the unit of sampling. If a long English article has ten thousand tokens and a short Chinese article has one hundred, taking a hundred documents from each produces a hundredfold difference in reading volume. We can organize each pool into equal-length samples with consistent counting rules, then mix them by quota. With variable lengths or padding, we track actual tokens and correct sampling. Megatron's blended dataset organizes reads by sample indices and weights; we still need to verify how those sample weights land in the token ledger.

Within each pool, we shuffle so similar material does not always arrive together. The Chinese allocation is 1,400B against a 500B inventory target, averaging 2.8 passes. Actual exposure for each record depends on sampling and the stopping point. The 2.8 is an initial sampling intensity, not a requirement to read every document the same number of times. Logs record cumulative tokens, unique documents covered, and repetition, with later changes decided by equal-budget experiments.

The 10T remains the workload ceiling for the from-scratch example. A route that reuses existing weights sets its own incremental quotas from the current model's capability gaps. Both routes use the same counting and recording methods.

Sampling weight: The relative share that determines how much content the training program draws from each category.

Random seed: A number controlling the starting point of a random process. Together with the same data, program, and sampling state, it can reproduce the same order.

What Evidence Should Change the Quotas?

Once quotas were in the program, model behavior could guide the next adjustment. When code performance looked weak, I kept two branches for comparison: A retained 15% code; B raised it to 20%, reducing general English by five percentage points and leaving everything else unchanged. Both start from the same checkpoint, process the same number of effective tokens, use the same training settings, and record GPU hours.

This experiment changes one thing: some opportunities to learn general English become opportunities to learn code. We then check code, Chinese, general English, and mathematics together. How much code improves, and whether general reading gets worse, determines whether the next training block should use B. The comparison therefore includes the cost of increasing one category.

To contain experiment costs, smaller models eliminate clearly unsuitable recipes before validation at the target model's stage checkpoints. Each round records result variation and preserves an independent final test set. Part Seven's comparison between new and repeated material fits into the same experiment framework.

NVIDIA's mid-training study, published on October 15, 2026, offers a useful reminder: proxy metrics on a base model may not reliably reflect performance after post-training. The team therefore used a fixed short SFT run to compare recipes. For the coding assistant we want to build, we can likewise run the same short post-training stage on each candidate and compare actual tasks. Falling training loss is one signal; improvement in the intended use determines whether a recipe deserves more training.

Quotas can change by stage. Each adjustment the team approves creates a new version, specifying the checkpoint and cumulative token count at which it takes effect. The quota already spent in each category stays in the ledger. The remaining plan is rearranged within the remaining budget.

Checkpoint: Files that save the model and training state at a particular time, used to compare, resume, or continue training.

SFT: Further training with example inputs and target answers so the model learns how to complete tasks as requested.

What Do We Hand to the Training Team?

Returning to the long tutorial, its text had become samples we could read, count, and resume. Files alone would not complete the handoff. I put five things into the delivery package:

Align the counting rules before the main run. Part Seven's 10T supports material and compute planning. During execution, record body tokens, special markers, padding positions, and positions contributing to loss separately. Feed the actual processing volume and throughput back into Part Two's calculation to obtain the revised schedule. Differences caused by end markers and boundary handling become numbers we can check here.

Before handing it over, two tiny documents test the entire path: text reconstruction, correct IDs, whether documents can read each other, and which positions contribute to loss. Then the program reads a short stretch, saves its state, and restarts. The next samples must match an uninterrupted run. Only after both checks does the accepted sorting tutorial become material the training program can reliably use.

The tutorial could finally travel the whole path into training: text becomes IDs, long documents are sliced, short ones packed, quotas select samples, and the reading state is saved. A resumed run can pick up where it should.

Data can now arrive reliably. Looking back at the lessons themselves, another gap remains: some source texts give a conclusion without explaining how it was reached. Part Ten begins there, asking an existing model to help supply the missing process.