[Training a 500B LLM from Scratch] 7/25: Where Does the Training Data Come From?

Last time, the model read “It was raining outside, so I picked up my umbrella.” It had a correct answer and a direction for adjusting parameters. That small step worked. But the data page of the training plan still had ten trillion tokens waiting.

I needed enough Chinese, English, and code for this 500B model. At first glance, public dataset sizes looked impressive. Collections measured in tens of trillions of tokens were easy to find. Combining them into one training library exposed the real questions: what could we use, what overlapped, and how much would survive cleaning?

The 10T in Part Two counts cumulative reading by the model. How much to prepare on disk needs a separate account. I divided the ledger into two columns: distinct material actually retained after deduplication, and the reading allocated to it during training. Sources, quality, and repetition all had to make sense between those columns.

This run continues with a 500B model trained from scratch. Existing models enter the plan as helpers first: selecting material and writing teaching documents. Inheriting existing weights is another product route, compared separately in an expanded section below.

Find Data by Purpose, Then Build the Inventory

Facing the long dataset list, I first wrote down what the model should learn, then searched for material by capability. Chinese and English each need their own text sources; code needs programs, tutorials, and explanations of solutions. Later evaluations tell us which capabilities need more material. Text collections for pretraining, demonstrations for post-training, and environments for reinforcement learning serve different purposes.

The first list included FineWeb, FineWeb2, The Stack v2, FineMath, and arXiv. After checking dataset cards on October 19, 2026, the following additional sources can address different gaps. The first five rows supply pretraining material; the final two provide worked examples for post-training reference. We keep inventory for each stage separate.

For specialized CUDA capabilities, add Nemotron-SFT-CUDA-v1, released in 2026. It retains solutions that pass program tests and associated checks, making it useful for a specific code-optimization goal.

Read more: dataset sizes, access methods, and overlaps

Among the original sources, FineWeb reports more than 18.5T publisher-counted tokens. FineWeb2 lists roughly 543.5 billion words in its Chinese subset, which we need to tokenize ourselves. The full training version of The Stack v2 contains about 900B publisher-counted tokens. Hugging Face primarily provides identifiers, and the content is retrieved through Software Heritage. FineMath-3+ contains about 34B, and the accompanying InfiMM-WebMath-3+ about 20.5B. They overlap, and each selected 4+ collection is a subset of its corresponding collection. Access starts with FineWeb, FineWeb2, The Stack v2, FineMath, and the arXiv bulk access instructions.

IndustryCorpus2 lists roughly 1TB of Chinese and 2.2TB of English. Those byte counts need to be converted into project tokens. Nemotron-CC-v2.1 adds about 2.5T publisher-counted tokens. The original-code portion of Code-v2 contains about 340B. Math-v1's 3plus contains about 133B, including the roughly 52B in 4plus. FineWeb-Edu contains about 1.3T and is already part of FineWeb. All these published sizes must pass our provenance checks, cross-collection deduplication, and common tokenization before entering the project inventory.

OpenThoughts3's 1.2 million records come from about 75,000 questions, each with multiple traces. OpenR1-Math also provides multiple answers per question. Store record counts, unique question counts, and token counts separately so we can see the actual breadth of coverage. Record the teacher version alongside each example. Older teacher data can provide a reproducible baseline, while student experiments measure the additional value of newer teachers.

The NVIDIA pretraining datasets above use its training-data access agreement. IndustryCorpus2 also has access conditions. Check both the dataset license and original-source requirements, then retrieve content according to the documentation once access is granted. Selecting these sources does not mean downloading everything and mixing it into one collection.

The size in a catalog could not enter the inventory ledger yet. I started with a sample: check provenance and usage conditions, extract the text, recount with the project tokenizer, then measure cleaning speed and retention. Only after the sample passed would retrieval expand. A “large dataset” could gradually become a collection of usable material.

Putting samples together revealed the same page, code, or question arriving through several entrances. Inventory therefore needs cross-source deduplication. Assign subject-specific material to its subject and code to the code collection; put the remainder in general language. Add separate labels for language, length, and source date. A Chinese mathematics handout can carry three labels while appearing only once in the inventory.

Token: a small piece of text processed by the model. Different tokenizers produce different counts, so all project inventory is counted using the student's tokenizer.

Deduplication: identifying identical or highly similar material so the same content is not added to inventory through different sources.

Accepted inventory: the total material available for a training stage after provenance, quality, and deduplication checks.

Pretraining: Learning patterns in language, knowledge, and code from large collections of text to build foundational capabilities.

Post-training: Further training of a base model to help it answer questions or complete tasks as requested.

How Does the From-Scratch Plan Add Up to 10T?

With material assigned a place, I returned to whether it could support this run. Part Two selected a 10T initial pretraining budget for a 500B dense model. It uses the empirical starting point of roughly 20 training tokens per parameter to give the compute and cash plans a common workload.

500B × 20 = 10,000B tokens = 10T tokens

10T is the total distance this learning run will travel. How much distinct material to prepare depends on quality, coverage, and the value of rereading.

I set the data team’s initial delivery target at roughly 5T of accepted unique material, then allocated 10T of cumulative reading. Each batch must pass retrieval, licensing, quality, and cross-source deduplication checks. Only a count from the fixed tokenizer can enter this recipe:

6,160 + 790 + 1,400 + 500 + 990 + 150 + 10 = 10,000B = 10T

Original general English material accounts for 61.6% of cumulative training, public synthetic rewrites for 7.9%, general Chinese for 14%, code-related material for 15%, and mathematics and science for 1.5%. These initial shares combine the bilingual and code goals with the amount of unique material to prepare. For example, allocating 1,400B to a 500B Chinese inventory gives an average of 2.8 passes. That connects inventory with planned practice; the equal-budget experiments in the next section determine whether the allocation is worthwhile.

A 15% code share means 1.5T cumulative tokens. That needs another split, or “find code” could return only source files. I divided it into 500B of code-related web pages, 990B of source code, and 10B of teacher explanations. Their unique inventory targets are 250B, 490B, and 10B. Teacher explanations retain the original 10B pilot, which now accounts for 0.1% of the 10T total.

Based on public material checked on October 27, 2026, the team can begin preparing this code material with code-related web text from Nemotron-CC-Code-v1, retrieve source files through the metadata in Nemotron-Pretraining-Code-v2, and supplement them with L2 source code from UltraData-Code. The first includes technical explanations and tutorials; source-code metadata still requires file retrieval. Synthetic textbooks keep separate source records.

Candidate collections may contain the same repository code, and code-related web pages may also occur in the English material. After cross-source deduplication, each item belongs to one inventory pool, with other language and subject attributes retained as tags. Publishers' totals help identify candidate sources. The 490B source-code target is accepted only after retrieval, filtering, and counting with the common tokenizer. If inventory falls short, first add suitable sources, then use experiments to decide quota changes.

At the bottom, the columns total about 5T and 10T, an overall average of two passes. Yet Chinese has 2.8 passes and synthetic rewrites about 1.32. Those differences deserve examination. They begin as settings for small-model comparisons. The next question is whether spending the same computation on rereading is worthwhile. If training meets its goals early, keep the best checkpoint; 10T remains this round’s workload ceiling.

Continued pretraining: inheriting existing parameters and continuing to read material and predict the next token to develop domain knowledge or capabilities.

Supervised fine-tuning (SFT): further training on inputs and demonstration responses so an existing model can better complete tasks as requested.

Reinforcement learning (RL): updating parameters using the model's own attempts and feedback on their outcomes. Code tasks can use program tests to provide feedback.

Should the Next Compute Budget Go to New Material or Repetition?

I gave the recipe a starting order: new material that meets the quality standard and covers the target skills comes first. Repetition increases when a category runs short or selected material is worth revisiting. After each fixed amount of work, compare where the next compute budget belongs. “New” means the model has not read it in this run.

What Is a Repeated Token Worth?, released on October 4, 2026, compares these two investments. In its experiments with 14M to 2B models, fresh material was generally more valuable at comparable quality under a fixed compute budget. When material was limited, a second pass still provided substantial gains, with diminishing returns from further repetition. Selecting Repetition Counts Across Model Scales, released the same day, shows that the best count changes with model size. These two preprints support our ordering. Project validation determines the actual allocation for 500B.

A recent production report illustrates this allocation in practice. In NVIDIA's October 2026 plan for 500B tokens of midtraining, different sources had different reading counts within the same stage:

That actual recipe made the logic clearer: passes follow the task. One model can read less of one category and more of another. I would likewise allocate the required training volume first, then divide by inventory to obtain passes.

A reliable comparison needs a fair exam. I separated the material: the training collection supports learning; validation selects recipes, checkpoints, and stopping points; the final test set remains sealed until the plan is settled.

Before comparing, set a GPU-hour limit for the experiment, then convert it to a token allowance using measured throughput. Screen options within that allowance, then validate them at training checkpoints of the target model. Repeated experiments or estimates of sampling error help determine whether the observed improvement is reliable. Each branch starts from the same checkpoint with matched additional tokens and training settings, allowing us to attribute the difference to the material.

If continued pretraining aims to improve the model's eventual post-trained behavior, evaluation needs one more step. The October 2026 NVIDIA report found that some base-model proxy metrics did not reliably track final performance. It therefore used the same short supervised fine-tuning procedure to compare midtraining checkpoints. We also fix that step and compare the tasks we ultimately care about, rather than measuring only a reduction in text-prediction error.

That gives us a way to revisit the 2.8 passes in the table. After allocating training volume, recalculate:

Effective epochs for a category = training tokens actually allocated to that category ÷ accepted inventory tokens in that category

For the same 500B Chinese inventory, allocating 1,400B gives an average of 2.8 passes; allocating 1,000B gives two. Reaching the goal early can mean less than one pass. The table's 2.8 passes form an initial sampling plan. The data team first compares fresh material with selected repeats at equal cost before allocating repetition. English, code, and teacher samples are calculated separately.

Supervised fine-tuning is counted by question-and-answer examples. First check the task-solving gains from a segment of training, then decide whether to keep reading the same demonstrations. Its example count, response loss, and objective differ from broad pretraining, so its epochs are planned separately. Reinforcement learning also produces new attempts. Record questions used, answers generated, and updates made separately.

Checkpoint: a set of model parameters saved during training, used for evaluation, recovery, and comparison of subsequent plans.

Validation set: independent material used to adjust the training plan. It never enters training.

Final test set: material used for acceptance after the plan is settled. It remains sealed during tuning.

Effective epochs: cumulative training volume for a category divided by its accepted inventory. With mixed sampling, this can be fractional.

Why Collect CoT?

One small line still needed separate attention: 10B of teacher explanations, 0.1% of 10T. Existing material often supplies an answer without making its derivation clear. I wanted the student to see those steps, so existing models would write candidate lessons. They are the teachers here. The student remains our 500B model trained from scratch.

Start with a task we can check: “Sort a list of integers in ascending order and preserve duplicates.” For input [3, 1, 3], the result should be [1, 3, 3]. The teacher can provide just the answer, or explain the key choice.

Task: sort a list of integers in ascending order and preserve duplicates.

Key step: both 3s must remain. set(xs) merges duplicate values, so use sorted(xs) directly on the original list.

Check: the result [1, 3, 3] is sorted and retains all the original elements.

Answer: return sorted(xs).

“Both 3s must remain” adds a connection beyond the answer alone: the requirement changes the program choice, and the result needs checking. These intermediate solution steps are Chain of Thought, or CoT. A new task that also preserves duplicates gives the student a chance to reuse that method.

Where the explanation enters training changes what the student learns. I separated the uses in the plan:

Front-Loading Reasoning, accepted at ICLR 2026, studied early exposure to reasoning material during an 8B model's pretraining and observed gains in foundational capabilities and subsequent fine-tuning. NVIDIA's September 2026 code report used teacher traces directly for supervised fine-tuning of existing models, then added reinforcement learning according to experimental results. The two studies address different stages.

This round first produces 10B of code explanations as a pretraining-material pilot, comparing them with answer-only material. Validation gains determine whether production grows. A project using existing weights can first use CoT in SFT; if evaluations show that foundational capabilities need more work, it can then study reasoning documents for continued pretraining.

The CoT we collect is the teacher's visible solution text. Research shows that it may not fully reflect the factors influencing the model internally. The data team treats it as teaching material: does the process hold up, support the answer, and help the student solve new questions?

Fluent explanations can still be wrong. Production therefore begins with questions, reference solutions, and independent tests, followed by teacher-generated candidates. Once code passes tests, inspect the explanation step by step. Remove repetition and irrelevant content while retaining the conditions that determine the solution, necessary derivations, and checks. Validate again after editing. Deduplicate questions, solutions, source repositories, and evaluation-question families together so rewrites of the same question do not fill the training set.

Following the sorting task, I added empty lists, negative numbers, duplicates, and random lists to the tests. sorted(set(xs)) fails the duplicate-value test. Even if the code is correct, an explanation claiming “remove duplicates first” needs rewriting. Choose among “answer only,” “key steps plus answer,” and a longer process by comparing the same student at the same training-token budget.

Chain of Thought (CoT): intermediate solution steps written by a model, which can include analysis, derivation, checking, and correction.

Synthetic data: material generated by a model or program. Here we use verified teacher demonstrations.

Loss: a number measuring the distance between predictions and training targets. The training plan determines which positions contribute to it.

Teacher model: a model that provides demonstrations or other learning signals to a student.

Distillation: training a student using a teacher's outputs, probabilities, or other signals. The teacher can be larger than the student and can use a different architecture.

Who Should Write These Explanations?

With checks for the pilot lessons ready, the next choice is the teacher. Version updates belong in that comparison. The official GLM-5.3 model card says it uses the same base model as 5.2, with improvements coming from post-training. This illustrates the value of inheriting weights: preserve the foundation, then focus on improving how the model solves problems and carries out tasks.

For the first round of code teaching material, I give GLM-5.3 and DeepSeek-V4.1-Flash-1017 the same questions and tests. GLM-5.2 already has an NVIDIA code-distillation comparison we can consult. The comparison between 5.3 and 1017 depends on how the same student performs after learning from each. The lighter GLM-5.3-Flash can be a separate cost candidate, with 320B total and 18B active parameters representing a different architecture and scale from this article's dense 500B student.

Recheck the license when versions change. GLM-5.3 uses its own model license, so we cannot carry over the MIT label from 5.2. The 1017 weights use MIT. Production selection considers license suitability, the cost of accepted explanations, and the student's performance on new questions. Teacher generation costs get a separate line in the budget. The student only needs to learn from the final selected material.

How Would We Use Claude as a Teacher?

After adding open teachers to the list, I also wanted to compare Claude. Its API can supply demonstrations, but production needs permission to use the output for this training purpose and stable usage conditions.

As of October 19, 2026, section D.4 of Anthropic's Commercial Terms requires express approval to use its services to train competing models. We plan to train a general-purpose 500B student, so we first explain the use to its business team and obtain approval covering it. An existing API account establishes that we can call the service.

The cooperation proposal needs to describe the student's size and commercial release plan, the Chinese and code task scope, expected request and token volume, the answers, code, explanations, or visible reasoning summaries we intend to retain, and their training and redistribution uses. The official sales channel receives this request. Acceptance, pricing, and quotas depend on Anthropic's response.

After approval, generate material within the agreed model version and output scope:

1. The data team prepares questions, reference answers, and independent tests.
2. The approved API generates answers and verifiable solution explanations. Where authorization covers it, also retain the reasoning summaries supplied by the interface.
3. Independently check code, answers, and steps; deduplicate by question family; record source and license versions.
4. Run an SFT comparison against open-teacher material using the same student and budget, then decide the production share.

Claude's official thinking-interface documentation distinguishes returned summaries from full internal reasoning. The student can learn from authorized answers, explanations, and summaries. Verification and training results determine their quality. Until cooperation terms are in place, production uses self-hosted weights with a suitable license.

Read more: how does Anthropic limit unauthorized distillation?

Measures Anthropic published on October 12, 2026 include displaying reasoning summaries, protecting raw reasoning records, detecting extraction behavior with two-stage classifiers, and limiting or banning accounts that repeatedly violate the rules. These measures reduce the information available for collection and increase the cost of unauthorized collection.

Approved cooperation uses the agreed output and usage channels, and the data team records material according to the contract. If we provide a model service in the future, we will also need to decide which outputs to expose, which uses require separate permission, and how to identify unusual requests and handle appeals.

Read more: counting and costing the from-scratch route's 10B of teacher material

Suppose each accepted teaching document averages 2,000 student tokens across its question, retained steps, code, and training format. Then 10B corresponds to:

5,000,000 documents × 2,000 tokens = 10,000,000,000 tokens = 10B

That is five million accepted documents. The 2,000 is an average-length target for production planning. The final ledger adds up each document's actual token count. Near-identical questions that only change numbers or wording are merged during deduplication. We need coverage of tasks and solution methods.

The teacher must generate more candidates because some answers will fail checks. First produce 10,000 candidates. Record the retained document count and token volume, all generation and testing costs, and the student's small-scale comparison results. Use those measurements to calculate:

Estimated full production cost = 10B ÷ accepted tokens in the pilot × total pilot cost

CoT also belongs in the ledger. Suppose the teacher generates 5,000 teacher tokens for one candidate, and the edited document delivered to the student contains 2,000 student tokens. Generation cost follows the work the teacher actually performed. Student inventory records only the final 2,000. Removed steps and rejected candidates have already consumed teacher compute.

This cost includes rejected candidates, teacher input processing and output generation, tests, verification, and storage. Update it with actual throughput and acceptance rates as production scales. The teacher has its own tokenizer. Service billing or inference throughput uses the teacher's count; material entering the student ledger is counted again with the student's tokenizer.

The 10B sits inside the 10T capacity budget. Experimental gains determine the final production volume. If the full 10T workload is completed, the student's main pretraining computation follows the baseline in Part Two. If it finishes earlier, cost follows the actual processed volume. Teacher production requires additional compute, and verification requires additional engineering. List both separately in the data-preparation budget and timeline. The 76 days cover the student's main training window for the original 10T workload. Teacher generation is scheduled before the corresponding material must be delivered.

After the base model is complete, we will also use teachers to prepare instruction-and-answer examples so it learns to respond as requested. This belongs to post-training, with a separate record of example count, epochs, and cost. Even if a document is reused then, the student's additional reading creates a new training workload.

Compare the alternative: when should we inherit an existing model, and which student should we choose?

Let Existing Models Take the Test First

Our goal is Chinese, English, and code. My first step would be to select a few usable models and test them on tasks the company actually cares about. Are their Chinese explanations accurate? Can they understand English reference material? Does their code pass tests? Can they find information in a long document? Each goal needs a task with a clear acceptance criterion. We also record speed, GPU memory, and deployment cost.

We can directly inherit capabilities that already meet the requirements. For knowledge updates, we can first try retrieval, putting current reference material in front of the model. Training follows when we identify a capability that needs to be learned in the parameters. This tells us where the next dollar should go.

Gap we find	Next step	Material to prepare
Capabilities already meet product requirements	Deploy and evaluate; additional training can be zero	Keep an independent evaluation set and collect real usage feedback
Limited understanding of domain language, knowledge, or document structures	Continue pretraining, adding volume as gains justify it	Domain text, code, documentation, and material that preserves general capabilities
Knowledge is mostly present, but the model struggles to solve tasks or use tools as requested	Use supervised fine-tuning to learn from good demonstrations	Questions, tool interactions, verified steps, and answers
The model can attempt tasks, but its success rate on verifiable tasks remains too low	Experiment with reinforcement learning using outcome feedback	New tasks, test environments, and reliable outcome feedback

These stages can be combined. Continued pretraining strengthens the foundation; post-training organizes those capabilities into useful behavior. Existing weights reduce the work we have to repeat, although substantial additional training may still be useful.

NVIDIA's October 15, 2026 report, Priming Base Models for Agentic Post-Training, provides a concrete example. Two existing base models received another 500B and 280B tokens of midtraining before post-training. This is much less than repeating an entire pretraining run, while still being a substantial project of its own. Our budget follows our own capability gaps and learning curves.

When is training from scratch worthwhile? We might want to try an architecture that existing weights cannot support, redesign the tokenizer, control the provenance of all training material, or address important tasks where existing models consistently fall short and we have the resources to rebuild the foundation. Starting from scratch provides this design freedom. It also means paying again for broad learning and training stability. The final result still needs to be compared with existing models.

This series keeps the from-scratch route so we can explain the engineering from beginning to end. For a company building a product today, I would first establish a baseline with existing models, then decide which additional training is needed.

Baseline: a record of the model's capability and cost on the same tasks before additional training begins.

Which Existing Models Could Be Students?

The student is the model we will ultimately deploy. The teacher provides demonstrations. They can come from different teams and use different architectures. When choosing a student, first check the weights, training stage, license, and operating cost, then compare capabilities.

As of October 19, 2026, I would organize the first student evaluation as follows:

Student candidate	Size and stage	What to examine first	What the choice must account for
Qwen3.8-Flash-Next	125B backbone, plus 51B ngram and 4B MTP; roughly 180B in language components, with vision and other components counted separately; 6B active in the backbone; already post-trained	Current Chinese, English, code, and multimodal capabilities, as a smaller existing-weight baseline	Training support for the new hybrid architecture; Qwen Community 1.0 requires separate permission for commercial uses such as MaaS and AI work assistants
Laguna-M.1-base	225B total parameters, 23B active; pretraining only; Apache 2.0	Code-oriented base weights with a clearly identified architecture and training stage	Chinese, general capabilities, instruction following, and tool behavior need separate evaluation and training
DeepSeek-V4-Flash-Base	284B total parameters, 13B active; base version; MIT	Another comparison for Chinese, English, and code foundations	Its architecture and scale differ from our dense 500B model; total parameters, active parameters, and deployment costs need separate calculations

For a Chinese, English, and code product, first test whether the latest Qwen already meets the task requirements and confirm the license for the product's use. Use Laguna as the code-oriented base-model comparison. If we need general base weights, add DeepSeek 284B. This first round can answer whether inherited behavior is sufficient or whether training from a base version is worthwhile.

ngram memory module: a memory component retrieved using consecutive token sequences. The model card lists it separately from the backbone parameters.

MTP: a multi-token prediction module that helps the model learn or propose several subsequent tokens. Its parameters are counted separately from the backbone.

MaaS: providing model capabilities to customers as an online service. A model license may set separate conditions for this use.

Another option, GLM-5.3-Flash can also serve as a student comparison. It has 320B total parameters and 18B active parameters, has completed post-training, and releases its weights under MIT. Its size difference from the 500B target also belongs in the deployment budget.

The latest version should enter the evaluation. Whether an upgrade is worthwhile also depends on whether it provides the capabilities we need, whether our training framework supports it, and whether the license fits the product. A newer post-trained model can be a starting point for further learning. If an experiment needs the original base weights, choose a model that explicitly releases a Base version.

Many existing models use MoE. Total parameters determine how many weights must be stored, while the parameters activated for each token affect part of the computation cost. Network communication, memory modules, and implementation also affect speed. The earlier GPU count and training time for a 500B dense model must be recalculated for this new starting point.

Base model: a model that has completed broad pretraining and can receive further training. An explicit Base label usually distinguishes it from a post-trained version.

What Should the Data Team Deliver Next?

The data page has finally grown from a list of dataset names into three deliverables: unique inventory accepted by source, cumulative reading allocations, and comparisons between new and repeated material. Each category carries provenance, version, deduplication records, training use, and student-token counts. Teacher material also carries generation and verification records.

I checked the two column headings once more: 5T is the unique inventory target to produce and accept; 10T is the cumulative workload budget. Material enters inventory first, then receives reading allocations based on capability gaps. New and selected old material compete at equal cost, and their gains determine repetition. When goals are met or further learning stops producing reliable gains, keep the best validation checkpoint and stop adding volume. Teacher production costs remain a separate account.

But what separates a downloaded article from “accepted inventory”? The next article opens the files and follows the cleaning, deduplication, and quality checks that guard that boundary.