[Training a 500B LLM from Scratch] 10/25: How Can Existing Models Help Us Write Training Material?

The material could now reach training through the pipeline from the previous article. I opened a few more documents and found that cleaning and slicing had both worked, yet the lessons still lacked something: the process.

Some articles went straight to the conclusion. Some programs occupied only a few lines. Readers had to supply the connections themselves. Our 500B model needs to learn those connections too. I wanted to add a batch of clear explanations and see whether the same training time could teach it more.

I started with a short passage: “I read 2 pages on the first day, 3 on the second, and 4 on the third. I read 9 pages in total.” The last sentence gave the answer and skipped the calculation.

Expand the process, and the passage becomes: “After the first day, the total is 2 pages. On the second day, add 3 to get 5. On the third day, add another 4 to get 9.” Both describe the same event. The second connects 2 to 5 and 5 to 9. Those connections are what the additional material should supply.

Writing every explanation by hand would take considerable time. I asked an already trained model to draft candidates. It supplies the material, so we call it the teacher. Our 500B model learns from it, so we call it the student. Source text goes to the teacher; an explanation comes back for checking and admission to the inventory. A final test on new problems asks whether the student improves under the same training budget.

The teacher's material is synthetic data. Training the student on content supplied by the teacher is one form of distillation. Natural language, mathematics, and code can all gain material this way. I followed the running page count through to a complete lesson and its dataset record.

Synthetic data: Training material generated by models or other tools. Here, we use an existing model to help write explanations, then check them.

Distillation: Training a student with learning signals such as a teacher's outputs. Here, we use verified teacher text.

Why does this material include code?

The three daily page counts form one sequence: 2, 3, 4. Their running totals form another: 2, 5, 9. Extend the record to a year, and adding every previous day from scratch becomes tedious. I wrote the operation as a small program: remember the total so far, add each new number, and save the current total.

The program performs the same operation that we just described in ordinary language. Code is also text, with stricter rules for expressing operations. We want a general-purpose model to explain ordinary passages, understand other people's programs, and help write programs. Our training material therefore includes code and documentation alongside Chinese and English text. The tokenizer converts all of it into tokens the model can read.

The code can be brief even when its process deserves an explanation. The teacher now has a specific job: expand suitable programs into lessons so the student sees the program, execution, and result together.

Material like this already appears in real pretraining recipes. The Domyn Edge dataset documentation, released on October 20, 2026, describes adding reasoning traces to most Python, Java, and C++ documents for pretraining. It turns the idea of adding a process alongside code into a data production task. We can borrow the approach and test it on our own tasks; that enhanced code is awaiting release once source attribution is complete.

Part Seven introduced synthetic data and distillation, reserving 10B tokens for these explanations in our 10T recipe, or 0.1% of the total. They count within the code allocation; original code and documentation have their own accepted inventory. Part Eight explained cleaning, and Part Nine turned material into training input. Now we turn this small allocation into lessons we can actually produce.

Token: A small unit of text produced by a tokenizer. Ordinary sentences and programs are both split into tokens and converted into IDs that the model reads.

How does the teacher turn a result into a lesson?

The assignment starts by stating the program's purpose. It accepts a sequence of numbers and returns the running total through each position. For 2, 3, 4, it returns 2, 5, 9. With no input numbers, the result is empty. Page counts help explain the operation; the same program handles other numbers that need accumulating.

I put those three actions into the function below: remember the total, add the new number, and save the new total. Expand it to see the program, or follow the changing numbers in the explanation that follows.

def running_totals(numbers):
    total = 0
    result = []
    for number in numbers:
        total += number
        result.append(total)
    return result

With the program ready, I wrote the teacher's assignment:

Turn this program into a short lesson. First explain the problem it solves. Then use the input 2, 3, 4 to show how the total and result change at each step. Finally, explain what happens when there are no input numbers. Keep the original program, and make every step of the explanation match its execution.

The requested material is now specific enough to follow step by step: expand the process left implicit in the program and explain it with numbers we can check. The lesson's prose can read like this:

Running totals

Given a sequence of numbers, return the running total through each position, including the current number. For example, reading 2, 3, and 4 pages on successive days gives running totals of 2, 5, and 9 pages.

Initially, the total is 0 and the result list is empty.

Read 2 and add it to the total to get 2. Save it, giving the result [2].

Read 3 and add it to the previous total of 2 to get 5. Save it, giving the result [2, 5].

Read 4 and add it to the previous total of 5 to get 9. Save it, giving the result [2, 5, 9].

Each step saves the total through the current position. With no input numbers, the loop runs zero times and returns an empty list.

The complete lesson inserts the original program from the expandable section after the page-count example, then continues with “Initially, the total is 0.” Program, input, explanation, and conclusion share one document, making the teacher's additions easy to inspect. To expand production, we use the teachers and access methods selected in Part Seven for the intended training use, choose permitted code and documentation, and write explanation requests for each category.

Once the lesson was written, I ran the program first. It had to return 2, 5, 9, and every running total in the explanation had to agree. Empty input gets a real run too. As the collection grows, negative numbers, zero, and repeated numbers provide further checks: the difference between adjacent running totals should equal the new number at the current position.

Passing the code tests still leaves the explanation to read. Confuse the current input with the running total once, and the lesson teaches the wrong relationship. Programs check the numbers; sampling and review check accuracy, clarity, and missing conditions. Both must agree before the lesson enters the accepted inventory.

This pattern of generating and then checking also applies to ordinary language material. In research released on October 19, 2026, NVIDIA and AI4Bharat translated curated English documents into Indian languages, assessed translation quality, repaired low-quality content, and retained source-document records. Their released synthetic corpus contains about 2.83T tokens across 22 Indian languages; that is the size of the corpus. The study then selects material, mixes it with natural text for continued pretraining, and tests its value through controlled experiments. Code requires checks on execution; translation requires checks that the original meaning survives. Each needs its own verification method.

After acceptance came another production trap: replacing 2, 3, 4 with different numbers could quickly produce a mountain of similar lessons. I directed expansion toward different programs, difficulty levels, and explanatory approaches. Part Eight's deduplication and source tracking continue here, with derivatives of the same source managed together.

What does the student actually read?

The lesson had passed its checks. Next came the dataset record. The student still reads a document titled “Running totals,” whose body contains the task, program, step-by-step explanation, and conclusion. The data team adds a tracking label for the source, teacher version, and checks.

I saved the body separately from that label so review records would stay out of training. In the record below, text contains the complete lesson and metadata contains its tracking information. The training reader sends only the body to the tokenizer. The model sees the program and explanation; source IDs and review records stay in the data system.

{
  "text": "Running totals\n\nGiven a sequence of numbers, return the running total through each position, including the current number. For example, reading 2, 3, and 4 pages on successive days gives running totals of 2, 5, and 9 pages.\n\nProgram:\ndef running_totals(numbers):\n    total = 0\n    result = []\n    for number in numbers:\n        total += number\n        result.append(total)\n    return result\n\nInitially, the total is 0 and the result list is empty.\n\nRead 2 and add it to the total to get 2. Save it, giving the result [2].\n\nRead 3 and add it to the previous total of 2 to get 5. Save it, giving the result [2, 5].\n\nRead 4 and add it to the previous total of 5 to get 9. Save it, giving the result [2, 5, 9].\n\nEach step saves the total through the current position. With no input numbers, the loop runs zero times and returns an empty list.",
  "metadata": {
    "record_id": "example-001",
    "source_id": "self-authored-running-totals-example",
    "teacher_version": "illustrative-output",
    "review_record": "illustrative-review-record",
    "family_id": "running-totals-example",
    "split": "train"
  }
}

Metadata: Information attached to a document, such as its source, teacher version, and review record. It helps the data team manage the material.

JSON: A text format for structured data. The record above stores the document body and its tracking information separately.

With the record format settled, we count the body using the student's tokenizer and add it to the inventory. Teacher-generated volume measures production cost. Accepted tokens actually read by the student measure training volume. Rejected material still cost money to produce, so both records matter.

The inventory's 10B allocation can now become a production task. At an average of 2,000 student tokens per accepted lesson, 10B means five million lessons: 10,000,000,000 divided by 2,000. Five million lessons require a continuing production pipeline. Accepted volume depends on length, pass rate, and deduplication. A small pilot establishes cost per million accepted tokens before we decide how to expand.

The mixture from Part Nine schedules when the student reads them. At the reserved share, every 10B training tokens includes 10 million tokens of teacher explanations. Across the full 10T plan, the explanations account for 10B, already included in the total. Teacher explanations are included in the total code allocation; compute used to produce the material is counted separately.

How does the model learn from this lesson?

Production and storage were connected. I followed the running-total lesson into the student's training. A person reads that the total changes from 2 to 5. The model splits the same text into tokens and predicts what comes next from what came before. It reads the input 2, 3, 4 and an initial total of 0; the program and explanation unfold under those conditions.

During training, the lesson supplies the actual text that follows. The lower the probability the model assigns to that text, the larger its prediction loss. The training software uses the loss to calculate how to adjust the parameters and updates the model's weights. When the next document arrives, learning continues with those updated weights.

The task, code, steps, and conclusion all belong to this pretraining document and all provide learning signals. Splitting long documents, packing short ones, and handling document boundaries follow Part Nine's rules. At this stage, we are training a general-purpose base model using complete documents. Teaching it to answer conversational requests will have a separate post-training plan.

The teacher's job here ends with supplying the material. During training, the student reads saved lessons. Running totals supply one set of relationships; many different lessons bring more forms of expression and operations. The next question is whether the student can work beyond these familiar documents.

Prediction loss: A measure of how much probability the model assigns to the actual next tokens. The training software uses it to adjust the parameters.

Weights: Parameters that the model reuses across different inputs. Training changes and saves them.

How do we know these explanations are worth producing?

Does a fuller explanation deserve the training time it occupies? I kept a direct comparison. Starting from the same checkpoint, two runs read the same number of additional tokens with the same training settings. One reads teacher explanations according to the recipe. The other replaces that portion with an equal volume of original code and documentation. Total code shares remain equal, as do the shares of Chinese, English, and mathematics.

After training, the student faces unfamiliar material. Can it explain new programs, follow new execution paths, or complete new coding tasks? Have Chinese, English, and mathematics changed? We are training a general-purpose model, so a local change must also be judged by its broader effects.

For that test to mean something, its problems are reserved before lesson production and separated from training by source and task family. A development validation set helps find problems and refine the material; the final test waits until the plan is fixed. Small models eliminate clearly ineffective production methods, followed by target-model checkpoint comparisons, with teacher production, verification, and student training costs recorded together.

The ICLR 2026 paper Front-Loading Reasoning, first posted in 2025, compares adding reasoning data at different stages using 8B models. It provides a basis for studying this kind of material during pretraining. Our 10B allocation is the portion reserved for this project's experiment. Our own results will determine whether to retain, change, or expand it.

Back at the passage that ended with “9 pages in total,” the missing process was now present, along with program checks, an inventory record, and a student comparison. We can follow that record to decide whether the added explanation earns its place. Teacher output measures production. Whether the same 500B student learns more determines what we should write next.

The data work now forms one continuous line: Part Seven found sources and quantities, Part Eight cleaned documents, Part Nine organized training sequences, and this article added verified teacher-written lessons. As those materials keep arriving, the other end needs somewhere to receive them. I returned to Part Three's equipment plan. Part Eleven places the servers in a physical facility, with racks, power, and cooling, so reading and training can continue.