[Training a 200B LLM from Scratch] 7/25: Where Does the Training Data Come From?

In the previous article, we followed one training step using “It was raining outside, so I picked up an umbrella.” The model found the answer in the material and adjusted its parameters. Before preparing the full collection of training material, we need to answer an earlier question: does this company actually need to train from scratch?

Open models have already read enormous amounts of material. Inheriting their weights also means inheriting the capabilities learned from that material. How much more data we need depends on how far the model is from our goal.

This article first chooses the student's starting point, then plans what it needs to learn, which material to read, and how much. The earlier 4T remains the example budget for training a 200B dense model from scratch. A plan that uses an existing model gets a separate budget for additional training.

Let Existing Models Take the Test First

Our goal is Chinese, English, and code. My first step would be to select a few usable models and test them on tasks the company actually cares about. Are their Chinese explanations accurate? Can they understand English reference material? Does their code pass tests? Can they find information in a long document? Each goal needs a task with a clear acceptance criterion. We also record speed, GPU memory, and deployment cost.

We can directly inherit capabilities that already meet the requirements. For knowledge updates, we can first try retrieval, putting current reference material in front of the model. Training follows when we identify a capability that needs to be learned in the parameters. This tells us where the next dollar should go.

These stages can be combined. Continued pretraining strengthens the foundation; post-training organizes those capabilities into useful behavior. Existing weights reduce the work we have to repeat, although substantial additional training may still be useful.

NVIDIA's October 15, 2026 report, Priming Base Models for Agentic Post-Training, provides a concrete example. Two existing base models received another 500B and 280B tokens of midtraining before post-training. This is much less than repeating an entire pretraining run, while still being a substantial project of its own. Our budget follows our own capability gaps and learning curves.

When is training from scratch worthwhile? We might want to try an architecture that existing weights cannot support, redesign the tokenizer, control the provenance of all training material, or address important tasks where existing models consistently fall short and we have the resources to rebuild the foundation. Starting from scratch provides this design freedom. It also means paying again for broad learning and training stability. The final result still needs to be compared with existing models.

This series keeps the from-scratch route so we can explain the engineering from beginning to end. For a company building a product today, I would first establish a baseline with existing models, then decide which additional training is needed.

Baseline: a record of the model's capability and cost on the same tasks before additional training begins.

Continued pretraining: inheriting existing parameters and continuing to read material and predict the next token to develop domain knowledge or capabilities.

Supervised fine-tuning (SFT): further training on inputs and demonstration responses so an existing model can better complete tasks as requested.

Reinforcement learning (RL): updating parameters using the model's own attempts and feedback on their outcomes. Code tasks can use program tests to provide feedback.

Choose the Student Before the Teacher

The student is the model we will ultimately deploy. The teacher provides demonstrations. They can come from different teams and use different architectures. When choosing a student, first check the weights, training stage, license, and operating cost, then compare capabilities.

As of October 19, 2026, I would organize the first student evaluation as follows:

For a Chinese, English, and code product, first test whether the latest Qwen already meets the task requirements and confirm the license for the product's use. Use Laguna as the code-oriented base-model comparison. If we need general base weights and accept a larger model, add DeepSeek 284B. This first round can answer whether inherited behavior is sufficient or whether training from a base version is worthwhile.

ngram memory module: a memory component retrieved using consecutive token sequences. The model card lists it separately from the backbone parameters.

MTP: a multi-token prediction module that helps the model learn or propose several subsequent tokens. Its parameters are counted separately from the backbone.

MaaS: providing model capabilities to customers as an online service. A model license may set separate conditions for this use.

If we allow still more capacity, GLM-5.3-Flash can also serve as a student comparison. It has 320B total parameters and 18B active parameters, has completed post-training, and releases its weights under MIT. Its size difference from the 200B target also belongs in the deployment budget.

The latest version should enter the evaluation. Whether an upgrade is worthwhile also depends on whether it provides the capabilities we need, whether our training framework supports it, and whether the license fits the product. A newer post-trained model can be a starting point for further learning. If an experiment needs the original base weights, choose a model that explicitly releases a Base version.

Teacher choices should also follow current versions. The official GLM-5.3 model card says it uses the same base model as 5.2, with improvements coming from post-training. This illustrates the value of inheriting weights: preserve the foundation, then focus on improving how the model solves problems and carries out tasks.

For the first round of code teaching material, I would give GLM-5.3 and DeepSeek-V4.1-Flash-1017 the same questions and tests. GLM-5.2 already has an NVIDIA code-distillation comparison we can consult. The comparison between 5.3 and 1017 depends on how the same student performs after learning from each. The lighter GLM-5.3-Flash can be a separate cost candidate, although its 320B total and 18B active parameters also exceed this article's 200B student target.

Recheck the license when versions change. GLM-5.3 uses its own model license, so we cannot carry over the MIT label from 5.2. The 1017 weights use MIT. Production selection considers license suitability, the cost of accepted explanations, and the student's performance on new questions. Teacher generation costs get a separate line in the budget. The student only needs to learn from the final selected material.

Many existing models use MoE. Total parameters determine how many weights must be stored, while the parameters activated for each token affect part of the computation cost. Network communication, memory modules, and implementation also affect speed. The earlier GPU count and training time for a 200B dense model must be recalculated for this new starting point.

Base model: a model that has completed broad pretraining and can receive further training. An explicit Base label usually distinguishes it from a post-trained version.

Teacher model: a model that provides demonstrations or other learning signals to a student.

Distillation: training a student using a teacher's outputs, probabilities, or other signals. The teacher can be larger than the student and can use a different architecture.

Find Data by Purpose, Then Build the Inventory

I would work backward from the capability gaps when choosing material. Weak Chinese calls for more Chinese sources. Weak code-editing ability calls for repositories, commits, and executable tasks. Text collections for pretraining, demonstrations for post-training, and environments for reinforcement learning serve different purposes.

The original FineWeb, FineWeb2, The Stack v2, FineMath, and arXiv remain in the plan. After checking their dataset cards on October 19, 2026, the following additional sources can address different gaps:

For specialized CUDA capabilities, add Nemotron-SFT-CUDA-v1, released in 2026. It retains solutions that pass program tests and associated checks, making it useful for a specific code-optimization goal.

Read more: dataset sizes, access methods, and overlaps

Among the original sources, FineWeb reports more than 18.5T publisher-counted tokens. FineWeb2 lists roughly 543.5 billion words in its Chinese subset, which we need to tokenize ourselves. The full training version of The Stack v2 contains about 900B publisher-counted tokens. Hugging Face primarily provides identifiers, and the content is retrieved through Software Heritage. FineMath-3+ contains about 34B, and the accompanying InfiMM-WebMath-3+ about 20.5B. They overlap, and each selected 4+ collection is a subset of its corresponding collection. Access starts with FineWeb, FineWeb2, The Stack v2, FineMath, and the arXiv bulk access instructions.

IndustryCorpus2 lists roughly 1TB of Chinese and 2.2TB of English. Those byte counts need to be converted into project tokens. Nemotron-CC-v2.1 adds about 2.5T publisher-counted tokens. The original-code portion of Code-v2 contains about 340B. Math-v1's 3plus contains about 133B, including the roughly 52B in 4plus. FineWeb-Edu contains about 1.3T and is already part of FineWeb. All these published sizes must pass our provenance checks, cross-collection deduplication, and common tokenization before entering the project inventory.

OpenThoughts3's 1.2 million records come from about 75,000 questions, each with multiple traces. OpenR1-Math also provides multiple answers per question. Store record counts, unique question counts, and token counts separately so we can see the actual breadth of coverage. Record the teacher version alongside each example. Older teacher data can provide a reproducible baseline, while student experiments measure the additional value of newer teachers.

The NVIDIA pretraining datasets above use its training-data access agreement. IndustryCorpus2 also has access conditions. Check both the dataset license and original-source requirements, then retrieve content according to the documentation once access is granted. Selecting these sources does not mean downloading everything and mixing it into one collection.

For each incoming source, begin with a sample. Check provenance and usage conditions, extract the main text, count it with the project's tokenizer, and measure cleaning speed and the retained fraction. Expand the download after confirming its value. Dataset names and public sizes help us find material. The acceptance ledger records what we actually retain.

Different sources often contain the same web page, code, or question, so we also need cross-source deduplication. Assign subject-specific material to its subject and code to the code collection; put the remainder in general language. Add separate labels for language, length, and source date. A Chinese mathematics handout can carry three labels while appearing only once in the inventory.

Token: a small piece of text processed by the model. Different tokenizers produce different counts, so all project inventory is counted using the student's tokenizer.

Deduplication: identifying identical or highly similar material so the same content is not added to inventory through different sources.

Accepted inventory: the total material available for a training stage after provenance, quality, and deduplication checks.

How Does the From-Scratch Plan Add Up to 4T?

Now return to this series' example. Part Two selected a 4T initial pretraining budget for a 200B dense model. It uses the empirical starting point of roughly 20 training tokens per parameter to give the compute and cash plans a common workload.

200B × 20 = 4,000B tokens = 4T tokens

If the company chooses existing weights, additional tokens are planned from the baseline and gaps described earlier. The ledger below serves the from-scratch example specifically.

The capacity example allocates 2,500B to general English, 800B to general Chinese, 600B to code, and 100B to mathematics and science. General language provides breadth; code and subject material provide focused practice. Their shares are 62.5%, 20%, 15%, and 2.5%. Chinese and English within code and subject material are counted separately through language labels.

A library with 100 books still contains 100 books after we read them twice. The training system has completed 200 books' worth of reading. Separating inventory from cumulative work gives us the following auditable budget example.

2,500 + 800 + 590 + 100 + 10 = 4,000B = 4T

The 600B for code includes 590B of original material and 10B of teacher material. This example represents 3.255T of inventory plus 0.745T of repeated reading. The 1s and 2s in the right column connect inventory capacity with workload to estimate storage, processing, and training costs. During execution, we mix and sample accepted material according to the target capabilities and validate periodically throughout training. Compute for every repeated reading is allocated after the comparison in the next section. The 4T remains this series' example workload and current budget ceiling. Training can finish earlier if the goal is reached.

Should the Next Compute Budget Go to New Material or Repetition?

I would give the data team a clear default order. First expose the model to new material that meets our quality standard and covers the target capabilities. When a category has limited supply, or selected material has a clear advantage, allocate repetition to it. Check progress after fixed amounts of training and give the next compute allocation to the option with greater returns. Here, “new material” means material not yet read in this additional training run. We check the existing model's public records for what it previously read.

What Is a Repeated Token Worth?, released on October 4, 2026, compares these two investments. In its experiments with 14M to 2B models, fresh material was generally more valuable at comparable quality under a fixed compute budget. When material was limited, a second pass still provided substantial gains, with diminishing returns from further repetition. Selecting Repetition Counts Across Model Scales, released the same day, shows that the best count changes with model size. These two preprints support our ordering. Project validation determines the actual allocation for 200B.

A recent production report illustrates this allocation in practice. In NVIDIA's October 2026 plan for 500B tokens of midtraining, different sources had different reading counts within the same stage:

This was that team's actual recipe. It shows that one model can read less of one category and more of another. Our project likewise allocates training volume according to the task, then calculates effective epochs from the ledger.

Putting this into practice requires three separate collections. The training collection is used for learning. The validation set selects the recipe, checkpoints, and stopping point. The final test set remains sealed until the plan is settled, when it provides an independent acceptance test.

First set a GPU-hour limit for this experiment, then convert it to a token allowance using measured throughput. Screen options within that allowance, then validate them at training checkpoints of the target model. Repeated experiments or estimates of sampling error help determine whether the observed improvement is reliable. Each branch starts from the same checkpoint with matched additional tokens and training settings, allowing us to attribute the difference to the material.

If continued pretraining aims to improve the model's eventual post-trained behavior, evaluation needs one more step. The October 2026 NVIDIA report found that some base-model proxy metrics did not reliably track final performance. It therefore used the same short supervised fine-tuning procedure to compare midtraining checkpoints. We also fix that step and compare the tasks we ultimately care about, rather than measuring only a reduction in text-prediction error.

“How many passes?” then becomes a result we can keep checking:

Effective epochs for a category = training tokens actually allocated to that category ÷ accepted inventory tokens in that category

For the same 400B Chinese inventory, an eventual allocation of 800B averages two passes. An allocation of 600B averages 1.5. Reaching the goal early can mean less than one pass. The two passes in the budget table are for capacity estimation. The data team first compares new material with selected earlier material at equal cost, then releases the repetition allocation. English, code, and teacher examples are calculated separately.

Supervised fine-tuning is counted by question-and-answer examples. First check the task-solving gains from a segment of training, then decide whether to keep reading the same demonstrations. Its example count, response loss, and objective differ from broad pretraining, so its epochs are planned separately. Reinforcement learning also produces new attempts. Record questions used, answers generated, and updates made separately.

Checkpoint: a set of model parameters saved during training, used for evaluation, recovery, and comparison of subsequent plans.

Validation set: independent material used to adjust the training plan. It never enters training.

Final test set: material used for acceptance after the plan is settled. It remains sealed during tuning.

Effective epochs: cumulative training volume for a category divided by its accepted inventory. With mixed sampling, this can be fractional.

Why Collect CoT?

Consider a small task: “Sort a list of integers in ascending order and preserve duplicates.” For input [3, 1, 3], the result should be [1, 3, 3]. The teacher can provide just the answer, or explain the key choice.

Task: sort a list of integers in ascending order and preserve duplicates.

Key step: both 3s must remain. set(xs) merges duplicate values, so use sorted(xs) directly on the original list.

Check: the result [1, 3, 3] is sorted and retains all the original elements.

Answer: return sorted(xs).

These intermediate solution steps are called Chain of Thought, or CoT. An answer teaches the student a relationship between an input and an output. The steps also show how a requirement affects the choice and how to check the result. On another task that requires preserving duplicates, the student has an opportunity to reuse that method.

CoT is useful in both training stages, with different learning objectives.

Front-Loading Reasoning, accepted at ICLR 2026, studied early exposure to reasoning material during an 8B model's pretraining and observed gains in foundational capabilities and subsequent fine-tuning. NVIDIA's September 2026 code report used teacher traces directly for supervised fine-tuning of existing models, then added reinforcement learning according to experimental results. The two studies address different stages.

For the existing-weights route, I would first invest CoT effort in SFT and compare it with answer-only demonstrations. If the baseline shows that foundational capabilities still need work, we can then study whether to add reasoning documents to continued pretraining. The from-scratch example initially reserves a 10B code-explanation experiment, 0.25% of 4T. This proportion defines our initial production trial. Validation gains determine whether it grows.

The CoT we collect is the teacher's visible solution text. Research shows that it may not fully reflect the factors influencing the model internally. The data team treats it as teaching material: does the process hold up, support the answer, and help the student solve new questions?

For production, first prepare questions, reference solutions, and independent tests, then have the teacher generate candidates. Once code passes tests, inspect the explanation step by step. Remove repetition and irrelevant content while retaining the conditions that determine the solution, necessary derivations, and checks. Validate again after editing. Deduplicate questions, solutions, source repositories, and evaluation-question families together so rewrites of the same question do not fill the training set.

The sorting task can be tested with empty lists, negative numbers, duplicates, and random lists. sorted(set(xs)) fails the duplicate-value test. Even if the code is correct, an explanation claiming “remove duplicates first” needs rewriting. Choose among “answer only,” “key steps plus answer,” and a longer process by comparing the same student at the same training-token budget.

Chain of Thought (CoT): intermediate solution steps written by a model, which can include analysis, derivation, checking, and correction.

Synthetic data: material generated by a model or program. Here we use verified teacher demonstrations.

Loss: a number measuring the distance between predictions and training targets. The training plan determines which positions contribute to it.

How Would We Use Claude as a Teacher?

Closed models can provide demonstrations through an API. We need permission to use those outputs for training, together with sufficiently reliable conditions for data production.

As of October 19, 2026, section D.4 of Anthropic's Commercial Terms requires express approval to use its services to train competing models. We plan to train a general-purpose 200B student, so we first explain the use to its business team and obtain approval covering it. An existing API account establishes that we can call the service.

I would prepare a proposal describing the student's size and commercial release plan, the Chinese and code task scope, expected request and token volume, the answers, code, explanations, or visible reasoning summaries we intend to retain, and their training and redistribution uses. The official sales channel receives this request. Acceptance, pricing, and quotas depend on Anthropic's response.

After approval, generate material within the agreed model version and output scope:

1. The data team prepares questions, reference answers, and independent tests.
2. The approved API generates answers and verifiable solution explanations. Where authorization covers it, also retain the reasoning summaries supplied by the interface.
3. Independently check code, answers, and steps; deduplicate by question family; record source and license versions.
4. Run an SFT comparison against open-teacher material using the same student and budget, then decide the production share.

Claude's official thinking-interface documentation distinguishes returned summaries from full internal reasoning. The student can learn from authorized answers, explanations, and summaries. Verification and training results determine their quality. Until cooperation terms are in place, production uses self-hosted weights with a suitable license.

Read more: how does Anthropic limit unauthorized distillation?

Measures Anthropic published on October 12, 2026 include displaying reasoning summaries, protecting raw reasoning records, detecting extraction behavior with two-stage classifiers, and limiting or banning accounts that repeatedly violate the rules. These measures reduce the information available for collection and increase the cost of unauthorized collection.

Approved cooperation uses the agreed output and usage channels, and the data team records material according to the contract. If we provide a model service in the future, we will also need to decide which outputs to expose, which uses require separate permission, and how to identify unusual requests and handle appeals.

Read more: counting and costing the from-scratch route's 10B of teacher material

Suppose each accepted teaching document averages 2,000 student tokens across its question, retained steps, code, and training format. Then 10B corresponds to:

5,000,000 documents × 2,000 tokens = 10,000,000,000 tokens = 10B

That is five million accepted documents. The 2,000 is an average-length target for production planning. The final ledger adds up each document's actual token count. Near-identical questions that only change numbers or wording are merged during deduplication. We need coverage of tasks and solution methods.

The teacher must generate more candidates because some answers will fail checks. First produce 10,000 candidates. Record the retained document count and token volume, all generation and testing costs, and the student's small-scale comparison results. Use those measurements to calculate:

Estimated full production cost = 10B ÷ accepted tokens in the pilot × total pilot cost

CoT also belongs in the ledger. Suppose the teacher generates 5,000 teacher tokens for one candidate, and the edited document delivered to the student contains 2,000 student tokens. Generation cost follows the work the teacher actually performed. Student inventory records only the final 2,000. Removed steps and rejected candidates have already consumed teacher compute.

This cost includes rejected candidates, teacher input processing and output generation, tests, verification, and storage. Update it with actual throughput and acceptance rates as production scales. The teacher has its own tokenizer. Service billing or inference throughput uses the teacher's count; material entering the student ledger is counted again with the student's tokenizer.

The 10B sits inside the 4T capacity budget. Experimental gains determine the final production volume. If the full 4T workload is completed, the student's main pretraining computation follows the baseline in Part Two. If it finishes earlier, cost follows the actual processed volume. Teacher production requires additional compute, and verification requires additional engineering. List both separately in the data-preparation budget and timeline. The 31 days cover the student's main training window for the original 4T workload. Teacher generation is scheduled before the corresponding material must be delivered.

After the base model is complete, we will also use teachers to prepare instruction-and-answer examples so it learns to respond as requested. This belongs to post-training, with a separate record of example count, epochs, and cost. Even if a document is reused then, the student's additional reading creates a new training workload.

What Should the Data Team Deliver Next?

For a company preparing a product, the first deliverable is a capability and cost baseline for existing models. The second is a training plan matched to the gaps: add knowledge, learn demonstrations, or practice verifiable tasks. Each material category enters inventory with its source, version, deduplication record, training purpose, and student-token count. Teacher material also carries generation and verification records. Additional data volume is approved incrementally according to checkpoint gains. Reading counts follow the actual allocation.

For this series' from-scratch example, 4T remains the workload budget, and 3.255T is the corresponding inventory-capacity example. The data team first accepts the actual inventory, mixes and samples according to target capabilities, and validates periodically throughout training. It compares gains from new and selected earlier material at a common cost before deciding whether to increase repetition. When the goal is reached or further learning brings no stable gain, save the checkpoint with the best validation performance and stop increasing volume. Teacher-production and student-training costs stay in separate ledger entries.

The next article starts with a web page and follows text extraction, cleaning, and deduplication. Whether we start from scratch or inherit weights, these steps determine whether the material we finally give the model is worth reading.