[Training a 500B LLM from Scratch] 8/25: How Do We Clean the Training Data?

As I brought together the sources selected in the previous article, a familiar problem appeared: the same tutorial had arrived through two entrances. One copy came from a general web collection; the other carried a curated label. Both were readable. The inventory could count them only once.

A few more files revealed different problems. Some were complete, some had lost half a conclusion, and others contained code that contradicted the explanation. The spreadsheet called for 5T accepted tokens. Each file in front of us still had to cross the same threshold.

Reading every document by hand would outrun the training calendar. I decided to reuse the preparation already done by the publishers, let programs handle clear-cut problems, and ask existing models to read the parts that required judgment. FineWeb-Edu already offers a precedent. Following one document through the process makes the division of work easier to see.

Start with What Is Already in the Warehouse

I opened the documentation for FineWeb first. Its publisher has already extracted the main text, filtered language and quality, and removed duplicates within crawl batches. We fix the version, inspect a small batch of records, and bring their text, URLs, dates, and existing scores into the project.

The approximately 5T unique inventory target in Part Seven must pass this process batch by batch. Code sources distinguish datasets that deliver text from metadata listing file locations and versions. The latter first require source-file retrieval, then enter the same cleaning and counting process. Documents appearing in both general English and code-related web collections count once after deduplication.

Existing quality labels stay with the text. Educational-value scores and mathematics-quality scores retain their names and scoring rules. Once sample checks confirm that they suit our material, they can help select it directly. The publisher's earlier work continues to earn its keep.

A museum visitor guide makes the threshold easy to see: “Open Tuesday through Sunday. Closed on Mondays.” If two websites reproduce the full guide, we retain one text and record both sources. Another capture ends at “Open Tuesday through Sunday. Monday.” We return to the source and recover the missing information before accepting it. Without that evidence, the record goes on hold. Preserve complete information, merge copies, and repair damage that we can verify: this is what cleaning looks like for ordinary prose.

I picked a sorting tutorial to follow through the remaining checks. Its task is to preserve duplicate elements: given [3, 1, 3], sorted(xs) returns [1, 3, 3]. Here, running the code helps verify the teaching material. Ordinary articles follow the same process, with source checks and human sampling doing more of the verification.

First came the duplicate from the opening: the tutorial could appear in both the general and curated collections. FineWeb-Edu is a subset of FineWeb, much like a library's full catalog and its recommended reading list. We keep both sources and the selection label, but count one document in the unique inventory.

Metadata: Information about the text, such as its source, version, license record, and quality labels. It can be stored separately from the text sent to the model.

Subset: A selection from a larger collection. Its material is already included in the larger collection.

Run the Inexpensive Checks First

Programs can first check for empty records, unreadable text, and visibly broken formatting. We also handle known subset relationships between sources here. FineMath-4+ is contained in FineMath-3+, so we record that relationship when selecting material.

When different sources copy the same content, text fingerprints can identify it. Shared passages help find near-duplicate versions whose titles differ but whose main text is the same. DataTrove provides these deduplication steps. After checking, we keep a complete representative version and record the other sources.

The sorting tutorial revealed the limit of textual similarity. “Keep duplicates” and “remove duplicates” differ by only a few words, yet they change the result. The latter uses sorted(set(xs)) and produces [1, 3]. Similar wording teaches a different task, so both belong in the collection.

That distinction also shaped the work I gave models. First, convert the text into vectors representing meaning and find similar candidates. Then compare their tasks, conditions, and answers. Similarity narrows the search; the retention decision depends on whether the material adds learning value. A single symbol in code or a single condition in a question can change the answer.

Filtering rules also depend on the type of material. Code needs punctuation and indentation, mathematics needs formulas, and a short, complete proof can be worth keeping. When sample checks find damage to a category, we adjust its rules. An August 2026 deduplication study from the Hunyuan team likewise separates duplicate detection from retention policy and protects code structure.

Content fingerprint: A short identifier calculated from content to help find identical material quickly.

Near duplicate: Material with largely the same main content but a different title or small changes in wording.

Vector: A sequence of numbers representing features of content. Here, we use it to find text with similar meanings.

What Can Already Trained Models Take Over?

The copies had been merged and broken formatting caught, but the surviving documents still varied in quality. A program could find a repeated sentence without knowing whether a tutorial explained its subject clearly. Judging purpose and quality was the next job for an existing model.

I started with small models that could score material directly. The FineWeb-Edu classifier is a public model with about 0.1B parameters that scores the educational value of English text. Its developers first used Llama3-70B-Instruct to score 450,000 documents, trained the small model on those examples, and used it to process text at scale. The large model helped establish the judgment; the small one handled the repetition.

These classifiers are already trained. A first pass over our samples shows whether their scores are useful, and we schedule bulk processing where suitable scores are missing. As of October 2026, the public-weight candidates include:

The official pages for all four sets of weights list the Apache 2.0 license. The first three are specialized classification tools; Qwen is a general model that can follow instructions. NVIDIA's quality classifier also integrates directly with the September 2026 release of NeMo Curator, so existing tools can form a batch-processing pipeline.

With scores in the spreadsheet, the next question was what a high score actually meant. FineWeb-Edu particularly favors basic and primary-to-secondary educational material. Research papers, mathematics, and code need separate calibration. An educational score asks whether the text resembles useful teaching material; a topic label says what it discusses. Code correctness needs execution. We sample and calibrate Chinese, code, and mathematics separately rather than letting an English educational score decide every document's fate.

Long-document checks exposed another easy oversight: how much had the scoring model actually read? Long texts need section-by-section reading and an aggregation method. NVIDIA's quality classifier has a context length of 1,024 tokens. Feed it only the opening, and the result describes only the opening. We record the sections read, sample whether key conditions and conclusions were covered, and choose the aggregation method accordingly.

Classifier: A model that reads content and returns a category or score, such as “computing topic” or “high educational value.”

Calibration: Using human-checked samples to see how model scores correspond to the material we want to keep, then setting selection criteria.

Let a Large Model Read the Sorting Tutorial

A simple score could not explain the conflict between code and instructions. I gave the tutorial to a model that could follow a more specific request: “Find the tutorial's requirements, code, and expected output. Determine whether they agree. Identify the relevant passages and recommend keeping, holding, or repairing the document.”

The complete version asks us to keep duplicate elements, uses sorted(xs), and gives [1, 3, 3] as the output. All three agree, so it can proceed to acceptance checks. If another record has the same requirement but uses sorted(set(xs)), the model should point out that this code removes duplicates.

We then run a test. Passing [3, 1, 3] to the code actually returns [1, 3], making the problem clear. We also check empty lists, negative numbers, and other cases relevant to the task. The model helps locate the error; independent checks determine whether the material can enter training.

The checks now gave each document one of three exits. Complete material that passed stayed. Material with missing or unverifiable conditions went on hold. Where the evidence supported a repair, we created a new version and checked it again.

When the original tutorial explicitly requires keeping duplicates, the repair has a basis: ask the model to replace sorted(set(xs)) with sorted(xs), then inspect the differences and test results. If the question itself is missing, a guess about what the author intended supplies no such basis. That record stays on hold.

The same requirement for evidence guides formatting repairs. We restore paragraph boundaries supported by the source, fix indentation that can be checked against the original file, and organize formulas we can compare with the source. Factual repairs return to trusted sources; mathematical derivations need their steps and conditions checked. Starting with valuable, readily verifiable repairs puts the processing budget where it can help.

Hold: Temporarily keep questionable material out of the training manifest while preserving the original text and issue record for later review.

Save Rewrites as Separate Versions

After the repairs, some useful material still read like scattered notes. A model can organize it into a clear explanation or questions and answers. NeMo Curator's Nemotron-CC workflow provides rewriting, knowledge extraction, and question-generation steps. There is already an implementation to build on.

When the rewrite came back, I saved it as a synthetic version beside the original. The source, model, processing instructions, and verification results gave every change a traceable starting point. After content checks, the new version still goes through duplicate checks and evaluation isolation.

For example, a model can turn the sorting tutorial into a set of questions and answers, but they still come from the same knowledge material. The original and its rewrite belong to one content family and are handled together when splitting training and test data. The new wording can be used in training experiments while its source relationship remains clear.

Looking back at the complete articles we received, those can simply pass through. Model work belongs first where scores are missing, damage is clear, or another form of expression serves a useful purpose.

Synthetic data: Material generated or rewritten by a model. The project separately records its source, generation process, and verification results.

What Does This Actually Save?

By this point, less material needed reading by hand, and fewer complex judgment rules needed writing from scratch. Small classifiers returned scores. Large models interpreted difficult requirements, explained problems, and helped build our own labeled examples. As those examples accumulate, a smaller classifier tailored to our collection can take over more bulk scoring.

But model-assisted reading has a bill of its own. A large model reads the text and may generate reasons and a revised document. With API billing, count all inputs, outputs, and retries. With local deployment, record GPU hours and completed material per hour. A representative pilot estimates the cost per million input tokens and how much of that material is worth keeping or repairing.

After looking at those costs, I settled on layers: reuse suitable scores, handle obvious problems with inexpensive rules, apply small classifiers to appropriate bulk material, and send difficult cases to large models. We sample all three outcomes, kept, held, and repaired, to check both the errors admitted and the good material excluded.

The cost sheet still could not tell us whether the student learned more. For that, we compare two runs with the same training budget: one using the existing selection, the other including the new cleaning results. We check gains in the target capabilities and regressions in Chinese, mathematics, and code before expanding processing. The cleaning pipeline earns its place through what the model learns.

Threshold: The cutoff used for selection. Changing it affects both the amount retained and the pattern of mistakes.

Who Does the Work, and How Many People Do We Need?

Once processing volumes and review queues went onto the calendar, the human work became clearer. Engineers maintain the pipeline; people who understand the material judge difficult cases. Following the team plan in Part Three, four internal data roles join as the project progresses:

These are the responsibilities of the four internal staff members. Before month 7, the first two arrivals maintain versions and counts, with the evaluation team helping validate quality. Month 2 establishes samples, standards, and trials. Batch processing begins in month 3, model-assisted processing expands in month 5, with month 7 as the target for full-batch acceptance. First estimate processing volume and retention by category against the 5T inventory target, then use measured batch throughput and expert hours to check the schedule. If delivery moves, reschedule the main training start with it. Each expansion depends on whether throughput, retention rate, and the human-review queue can keep up with the training schedule.

The four people manage automation and delivery; expert review and bulk annotation are purchased by project. I record pilot work time before calculating vendor staffing. We time each material type, second reviews, and rework. Pending volume multiplied by average handling time, plus review and rework hours, divided by each person's available hours before delivery, gives the staffing requirement. Chinese, code, and mathematics are counted separately. A difficult proof takes very different effort from ordinary prose.

Progress checks then follow the bottleneck. Slow machines call for better batching or more compute. An overflowing expert queue calls for more outside review. Persistent overload in quality rules and pipeline maintenance calls for another internal role. Staffing follows the work, and outsourced labor has its own quantity and cost alongside the four internal positions.

What Can We Buy Directly?

The review queue can be outsourced, and missing material can be purchased. I first separated four purchases: a licensed collection of articles, a batch of expert demonstrations, a review service, and software for running the cleaning pipeline. Each solves a different problem. The official product information available on October 19, 2026, gives us these starting points:

Scale's generative AI services are particularly suited to having programmers, language specialists, and domain experts create or review demonstrations. Its Data Engine page explicitly lists question-answer generation, preference feedback, and evaluation after initial pretraining. We therefore include it in procurement for high-value expert data, while bulk pretraining text continues to come from existing corpora and licensed material.

Defined.ai is closer to selecting an existing collection: browse the catalog, obtain samples, then confirm usage rights and pricing for the selected material. Purchased articles still need cross-source deduplication against our inventory. Labelbox's role here is expert review: we supply the material and standards and receive traceable review results.

These custom services require project quotes. Scale's quote entry point leads to sales, Defined.ai offers samples and quote requests, and Labelbox expert projects are also scoped to requirements. NeMo Curator is Apache 2.0 open-source software run by our own team. Separating software, model compute, and expert fees makes total costs comparable.

With the supplier list in place, I put pilot production and acceptance ahead of expansion. Procurement follows five steps:

1. Describe the missing material. Include examples to keep, exclude, and debate, along with the delivery format.
2. Ask the provider for a small pilot batch, including sources, collection or generation methods, usage rights, and processing records. Specify permitted model-training and commercial uses in the contract, including the permitted uses of delivered data and trained weights.
3. Have the internal team accept or reject the batch using check samples whose answers the supplier cannot see. Check content, duplication, evaluation leakage, and retention rate. Run code tests and arrange independent review for specialist judgments.
4. Price by accepted, unique material. For text, use cost per million accepted student-model tokens; for expert work, use cost per accepted demonstration or review. Pilot costs, rejected material, compute, rework, and transfer all belong in the numerator.
5. Put acceptance standards, batch sizes, delivery dates, review methods, and replacement responsibilities into the contract, then expand batch by batch. Keep traceable records for every batch, with final acceptance owned internally.

Data licensing and outside processing are quoted again against the new 5T inventory target and entered into Part Three's complete funding formula. Pilot results decide which gaps need purchased material and which can use existing data and automation. Batch quantities, accepted unit costs, and delivery dates then go into the contracts. A source with low retention, heavy duplication, or frequent expert corrections can cost more per usable item even when its initial quote looks inexpensive.

Preference labeling: Comparing two responses and recording which better meets the requirements and why, for training or evaluating response quality.

Cost per accepted unit: Total cost divided by the amount of unique material that passes acceptance. Units can be student-model tokens or complete tasks; comparisons use the same standard.

Preserve the Exam and Deliver an Auditable Training Set

As delivery approached, one final separation needed checking. Copies and rewrites of the same content must be grouped into families before assignment to training, validation, and testing. Validation selects the cleaning approach; the final test waits until that choice is fixed. Public benchmark questions, answers, and solutions get their own exclusion list, checked for exact matches and approximate leakage.

Once the selection is fixed, count it with a fixed version of the student model's tokenizer. A Chinese mathematics handout can have both “Chinese” and “mathematics” labels, but it counts only once in the total inventory. Record unique inventory separately from cumulative training reads, giving Part Seven's repetition counts a reliable denominator.

The training team ultimately receives four things:

I opened the sorting tutorial again. It now had one inventory record, its copied sources and curated label attached, and checks for agreement between the code and explanation. An existing model had read it, a program had run it, and a person had reviewed the difficult parts. Its origin and quality were now explainable.

The four-person internal data team, existing software and models, and expert services purchased for specific gaps formed one pipeline. Yet a complete, accepted article still cannot fit unchanged into every training window. The next problem is where to cut that tutorial and how to place its pieces in front of the model.

Duplicate family: A group of copies, republications, and related rewrites of the same content.

Evaluation contamination: Test content entering training material, causing scores to overstate the model's ability to handle new problems.

Tokenizer: A tool that converts text into the token sequence the model reads. Counts use a fixed version and configuration.