[Training a 500B LLM from Scratch] 3/25: What Is the Full Budget?

The last line on the purchase sheet read $302 million. It covered 640 main training servers, twenty spares, and even the training NICs. Yet the list still left the machines with nowhere to go, no way to communicate, and nobody responsible for making them run.

The 76 days calculated last time covered only the main run after 5,120 B200s had begun processing a dense 500B model and 10T tokens, about 75.4 days by the calculation. There was still a road to opening day. I added two columns beside the purchase list: when it could be used, and when payment was due. Server and switch prices were checked on October 27, 2026; other sources retain their own dates.

The task now was to turn servers on paper into a working system. I followed their route into the facility, added whatever was missing, and placed each expense on the calendar. The full budget would take shape along that route.

Before a Purchase This Large, Ask Why We Should Buy

At more than $300 million, I paused. For one run, would renting capacity and stopping afterward be better? A cloud inquiry went on the list too: 5,120 B200s available together for 76 continuous days, with the training network, availability date, exclusive-use scope, failure replacement, and billing conditions. A rented trial could then inform the later purchase.

The plan calls for repeated research runs, so the machines will continue working after the first one. I kept ownership as the main route and worked through one specific configuration. On October 27, 2026, the Exxact eight-B200 AMD configurator showed $457,798 after selecting eight 400G ConnectX-7 NICs. The configuration includes dual EPYC 9554 processors, 1,536GB of host memory, one 3.84TB SSD, eight B200 GPUs, and a three-year warranty. The NICs are already in that server price. The network table separately adds switches, optics, and fiber.

660 × $457,798 = $302,146,680

640 main servers would train on 5,120 active GPUs. Twenty spares would carry another 160 GPUs, a 3.125% server maintenance allowance. A spare would take the place of a failed machine. The full configuration, NIC part numbers, delivery batches, warranty, and acceptance criteria needed to travel with that list to the supplier.

Cluster: Several servers working together.

RFQ: A request for quotation, stating the requirements a supplier must price.

Which switches should we buy, and how do they connect?

With the server configuration priced, I turned to its rear panel. The eight GPUs inside already had high-speed connections, but a result from one server still had to reach another. One 400G training NIC per GPU gave 5,120 ports across the 640 main machines. Those ports needed roads between them.

I selected the NVIDIA Quantum-2 MQM9700-NS2F, with 64 logical 400G ports per switch. The Colfax public catalog shows $34,555 each. At this scale, use three layers: servers connect to leaves, leaves to local aggregation, and the core connects the groups.

To check the route, I followed one result through the diagram. It leaves a server, crosses its local leaf and aggregation layers, and travels through the core to another group of servers. Each layer reserves as much capacity upward as downward, giving the machines room to exchange results. This design needs 160 leaves, 160 aggregation switches, and 96 core switches, or 416 switches in total. At the public unit price, the switches cost $14,374,880.

NIC: A network interface card connecting a server to a network.

Switch: A device connecting several devices so they can exchange data.

Pod: In this design, 128 training servers and their local switches.

Leaf: The lower switch layer next to the servers.

Aggregation: The middle layer connecting a pod's leaves to the core.

400G: A nominal transmission rate of 400 billion bits per second per link.

Following that route, the switch counts line up: 160 leaves + 160 aggregation + 96 core = 416. Server-to-leaf, leaf-to-aggregation, and aggregation-to-core each have 5,120 connections, making 15,360 links. This balances the port and bandwidth budget. The integrator then configures routing using NVIDIA's support for parallel-link fabrics and validates cross-pod communication and fault recovery with the target training workload.

Core: The upper switch layer connecting different pods.

Topology diagram: A drawing of which devices connect to one another.

Once the ports balanced, cable length raised another question: could all these machines occupy one place? The initial layout used three halls on one campus, bringing distance into procurement. Hall A contains pods 1 and 2, Hall B pods 3 and 4, and Hall C pod 5 and the core. The four remote pods require 4,096 cross-hall links. The NVIDIA MMS4X00-NM dual-400G single-mode module is a candidate within its specified 500-metre reach, subject to actual routes and losses. Initially allocate the remaining 11,264 links to within-hall short paths, then change optics and fiber where a route exceeds multimode reach.

Check the calculation: how do we get 416 switches and the optics quantities?

Divide 640 servers into five pods of 128. Each pod has 32 leaves. Every leaf connects 32 NIC ports below and uses its other 32 ports to reach all 32 aggregation switches in that pod. Aggregation also uses 32 ports downward and 32 upward, giving equal capacity on both sides.

The core contains 32 groups of three switches, 96 in total. The j-th aggregation switch in every pod connects to the j-th core group, splitting its 32 uplinks into 11, 11, and 10. Across five pods, those core switches use 55, 55, and 50 ports, each within the 64-port limit.

There are 12,832 dual-port modules. Core switches using 55, 55, and 50 ports require 28, 28, and 25 modules, leaving 64 unused half-ports across the design. That remainder is included in procurement.

Part 12 will map the connections to physical slots and English handover records. For now, we take the machines and cables to a facility that can hold them.

The quantities of transceivers and fibers were established; final distances and compatible combinations would determine the quote. I left a place for that total, O. Shared storage, separate storage and management networks, and software support also received defined scopes. Each incoming price could then be matched to the part of the system it completed.

Optical transceiver: A component converting electrical signals to light and received light back to electrical signals.

Single-mode and multimode fiber: Different optical media. Reach, transceivers, and connectors must be selected as a compatible combination.

Which actual facilities should we ask?

To find out how many machines a facility can hold, I started with power. 660 servers at a 14.5kW design allowance require 9.57MW. The 416 switches add 0.71552MW at the official maximum of 1.720kW with active connections. Reserve another 0.5MW for storage, management, and other IT equipment. We ask the operator for three 4.5MW halls on one campus, 13.5MW in total, leaving 2.71448MW of design headroom.

With the power outline established, I placed the servers in cabinets. At two per cabinet, 660 servers require 330 compute cabinets. Halls A and B each hold 264 servers and 128 switches, about 4.04816MW each. Hall C holds 132 servers, 160 switches, and the 0.5MW auxiliary allowance, about 2.68920MW. Each hall's power allocation fits. The next step puts cabinets, cooling, floor loads, and optical routes into the operator's real floor plan.

kW: Kilowatt, a unit of power equal to 1,000 watts.

MW: Megawatt, a unit of power equal to 1,000kW.

kWh: Kilowatt-hour, the energy unit used in electricity billing.

Using these requirements to screen facilities, I put the first inquiry with Flexential Atlanta-Douglasville 2 in Georgia. It publishes 36MW of critical load, support for 80+kW cabinets, and high-density capability; its earlier hall specification was 4.5MW. Ask whether three halls can be delivered together, how long their optical routes are, and what the complete 13.5MW package costs. The August 2026 announcement still described the local 36MW project as under construction. The operator confirms actual hall assignments and access dates.

Relying on one facility would leave the start date dependent on one operator. Other candidates were screened against 13.5MW too. The supply information below was collected on October 18, 2026; capacity and delivery dates would need updating when we contacted them:

The facility list was ready, and the plan still needed a rent number. As of November 4, 2026, this plan uses $165 per kW per month, excluding electricity. CBRE's H1 2026 Atlanta report gives a $150–165 per kW-month asking range for 10MW-plus requirements. I choose the top of that range because this is a 13.5MW high-density AI deployment and the budget should not start from the lowest market figure. The $165 rate is a planning estimate, not a supplier quote.

The 13.5MW allocation equals 13,500kW. From Month 4 through Month 12, nine months of planned rent is:

13,500kW × $165/(kW·month) = $2,227,500/month

$2,227,500 × 9 = $20,047,500

The operator quote will replace this planning estimate. Negotiate deposits, minimum term, on-site service, and electricity billing together to make offers comparable.

After renting the capacity, actual equipment use would produce another bill. The operating budget uses 10MW of IT load, PUE 1.4, 730 hours per month, and $0.10 per kWh. A full operating month is:

10,000 × 1.4 × 730 × $0.10 = $1,022,000

Months 4 and 5 use half load; Months 6 through 12 use full load. That is eight full-month equivalents, or $8.176 million in Year 1. The 10MW figure is an operating budget; 13.5MW is leased capacity. Meter readings and contract terms update actual electricity cost.

PUE: Total facility energy divided by IT-equipment energy. PUE 1.4 means one unit used by equipment requires 1.4 units including cooling and other facility needs.

Which work fits colocation, self-build, and multiple sites?

With rent still a large blank cell, building a facility deserved comparison. I first checked the factor most likely to control the schedule: when power could arrive. JLL's 2026 outlook gives average power-connection waits above four years in major markets. Its roughly $11.3 million per MW construction figure is based on a 50MW air-cooled shell-and-core facility, excluding land and IT equipment. Our 13.5MW project would separately price land, design, power connection, construction, commissioning, facility staff, and long-term asset value.

That power-connection timeline kept the first-year plan on the colocation route: find powered capacity with a written delivery date, then align equipment arrival, hiring, and lease commencement.

Renting capacity in several places might assemble the required size sooner, but would stretch the main training network across longer distances. I kept data processing, independent experiments, and teacher generation available for separate sites. The main run would stay in three halls on the same campus, linked by dedicated training fiber, with measured throughput checked against the 76-day plan.

Colocation: Installing our servers in another company's facility and renting space, power, and cooling.

Latency: The time for data to reach its destination, which affects how long machines wait for results.

Price the twenty positions individually: who joins first?

Distributed training: Several machines cooperating on one training run.

Evaluation: Using test tasks to measure model performance.

Post-training: Further training of a base model to improve task performance and instruction following.

SRE: Site reliability engineering, which keeps systems running reliably through monitoring, incident response, and recovery.

M1: Project Month 1. M2, M3, and later labels indicate subsequent project months.

The machines and facility were on the sheet; people still had to make them work. I assigned the work needed for training, arriving at an initial engineering team of twenty: five in model work, four in data, three in distributed training, three in infrastructure, four in evaluation and post-training, plus a programme lead. The plan reuses existing training software and assigns facility operations and physical installation to external partners. More work, new algorithms, or weaker external support would require a different headcount.

Each role needed compensation capable of attracting someone. I started with public US recruiting bands, position by position. Training engineering roles list base pay of $200,000–290,000, GPU systems roles $200,000–300,000, and data engineering roles $182,000–242,000. Use these bands to build the budget, then set offers against each candidate’s responsibilities and experience. A frontier-lab chief scientist would receive a separately budgeted package. This package targets people with large-scale training or cluster experience who want technical ownership in an early team. The 20% lead bonus and 15% engineer bonus are our choices for responsibilities and cash planning. Candidates will also consider research scope, compute access, and the team, so discuss equity and technical decision-making authority together.

Hiring could not wait until the machines arrived. I put the programme and model leads first, using their readiness and the availability of initial funding as the start of M1, then worked backward to prepare for it. Those two positions receive pay in Month 1. The other four workstream leads target Month 2, with searches starting earlier. The large Month 2 purchase needs infrastructure and distributed-training leads to review the configuration; if those roles are unfilled, the decision moves. SHRM’s 2026 benchmark gives a median of 39 days to fill a non-executive position. That cross-industry measure ends at offer acceptance, so arrival and onboarding time must be added.

Base salary alone left the hiring account incomplete. Candidates would consider four parts together: base salary, performance bonus, sign-on payment, and equity. The six leads receive a target bonus of 20% of base; the other fourteen positions receive 15%. Pay the award in Month 12, prorated by months worked that year. Sign-on payments are $50,000 for leads and $25,000 for other positions, paid when they join. Keep the 25% employer payroll-cost and benefits allowance on base pay, then budget an additional 8% employer allowance on bonuses and sign-on payments. Benefits are counted once, and each additional cash payment has a place.

Adding each arrival month gives headcounts of 2, 6, 8, 10, 14, and 17, reaching 20 in Month 7. Year 1 includes $3.5703 million of base pay, $0.8926 million of existing employer costs and benefits, $0.6086 million of performance bonuses, $0.6500 million of sign-on payments, and $0.1007 million of incremental employer allowances. Total personnel cash is $5.8221 million, an increase of $1.3593 million over base pay and benefits alone. For example, an engineer joining in Month 7 on $245,000 base earns 245,000 × 15% × 6 ÷ 12 = $18,375 in first-year bonus. This lifts Month 12 team cash payments to $1.1571 million.

Beyond cash, early employees should share in the company’s long-term outcome. I reserved a 10% equity pool on a post-financing, fully diluted basis. Initial awards in the table total 4.35%, leaving 5.65% for later hires and additional awards. Vest over four years, with 25% at the first anniversary and monthly vesting thereafter. Equity attracts people through participation in future value. Its current cash cost is zero, while shareholders bear dilution. The first two leads are also employee positions in this schedule; founder ownership is determined separately. Carta’s compensation report published in 2026 found that from January 2024 to February 2026, median initial equity grants for AI/ML engineers grew 31%, compared with 9.1% growth in median salaries. That is another reason to build cash and equity into the recruiting package together.

Finally, I checked who would handle purchasing and everyday failures, so a complete technical roster would not leave work unowned. The programme lead approves procurement and payment; technical owners define specifications and acceptance. External legal, procurement, and finance specialists handle contracts, delivery responsibilities, tax records, and payroll services. Obtain an annual service quote for this procurement and staffing scope. The facility operates the building, power, and cooling; the integrator performs the contracted installation, cabling, and acceptance; the equipment supplier provides the contracted warranty. Three internal infrastructure engineers own configuration, monitoring, diagnosis, and recovery. Put round-the-clock onsite replacement and response into the external contract; the onsite service team provides the shift coverage.

Equity pool: Ownership set aside for employee incentives.

Fully diluted ownership: A share calculation that counts existing shares and rights such as options that could become shares.

Vesting: Earning the agreed rights over time.

Dilution: A reduction in existing owners’ percentage of the company when more shares are issued.

After funding, when can training begin?

As people arrive month by month, equipment needs a matching rhythm. Two deliveries of 330 servers, each with 320 active machines and ten spares, spread installation and integration work. Target procurement review in Month 2, deliveries in Months 4 and 6, full-scale acceptance in Month 7, and the start of 76-day main training in Month 8, continuing into Month 10. Take this sequence to the equipment supplier and facility operator to establish executable dates together.

Month 4 exposed another detail: half the machines does not mean half the core network or half the rent. Lease all three halls from Month 4, and deploy the shared core and storage in time for first-batch integration. Two deliveries divide the installation work. Deposits, shipping payments, and acceptance balances follow the purchase contract. Half the servers arriving does not halve every network component or the rent. The electricity plan starts with half operating load and then replaces that estimate with site measurements.

With equipment dates on the calendar, training still had to wait for material. I put data on the same timeline, starting with small samples in Months 1 and 2: measure retrieval, cleaning, deduplication, and tokenization; confirm permissions, inspect quality, and compare mixtures with small models. Part 7 sets a production target of roughly 5T qualifying unique tokens, allocated into 10T cumulative training tokens. Publisher totals do not add up directly to delivered inventory.

Once the pilot establishes production speed, schedule production in Months 3–6. Month 7 acceptance covers sources, deduplication, tokenizer, inventory, sampling allocations, and sustained data delivery. Main training starts after the last of equipment, facility, data, and training-team readiness. If a dependency slips, update installation, training, and contractual payments, then recalculate the cash gap.

Tokenizer: A tool deciding how text is divided into tokens.

Deduplication: Identifying and removing repeated material.

Evaluation contamination: Test questions or answers entering training data, making test scores overstate actual capability.

Bring equipment and operations into one budget

After following the machines’ route into operation, I returned to the original purchase sheet. Four items could now be added from public unit prices or selected calculation bases:

The current planning subtotal has reached about $350.57 million: servers and switches already reach about $316.52 million, first-year compensation and electricity add about $14.00 million, and planned rent adds $20.0475 million. Optics, storage, and services join the same ledger individually rather than disappearing inside the equipment price.

Let O be the optical-transceiver and fiber total. Let X include the remaining complete quotes: shared storage and separate networks, shipping and installation, insurance and taxes, data licensing and processing, teacher generation, software support, external site operations, legal and accounting services, and early cloud experiments. Each quote states delivery quantities and service periods. NICs and warranties already included in the server price are counted once.

The complete first-year calculation is therefore:

First-year procurement and operations = $350,567,156 + O + X

Every term in this total is in dollars. O and X remain the respective project totals awaiting quotes; the $350,567,156 planning subtotal includes nine months of rent at $165 per kW per month. Once all remaining quotes are assembled, replace the planning rent with the operator's price and add a 10% risk allowance to the complete budget to obtain the funding-coverage target. The total can change with quotes while preserving the reason for each change.

Looking again at the twenty-person team, its capacity depends on the handoffs: reuse training software, and buy physical installation and on-site shifts from external partners. The additional work from three halls and three network layers enters X through installation, operations, and response contracts. Track data processing against pilot throughput and progress in Months 3–7. If internal staffing must grow, identify the added role and start month, then update compensation.

Capital purchase: Spending to buy equipment and complete installation.

Operating expense: Recurring costs such as pay, rent, and electricity.

Equity funding: Money investors contribute in exchange for ownership.

Equipment financing: Debt used to acquire equipment, repaid with principal, interest, and fees under a contract.

Can a new company obtain equipment financing?

At this scale, could instalments relieve the early cash pressure? I first looked at the companies equipment lenders were willing to finance. A compute provider can point to customer contracts and recurring service revenue as the source of monthly payments. This company spends first to train its own model, with revenue coming later. A lender must assess shareholder capital, available cash, the credibility of future revenue, and recovery from selling the equipment. GPUs have resale value; the company still needs a credible source of repayment.

Data Sales advertises equipment lease lines of up to $50 million. Its Ori case study shows how such financing happens: Ori had spent five years developing its platform, acquired GPUs to fulfill customer contracts, began with a lease in the tens of millions of dollars, and expanded the relationship as the business grew. Customer demand and an operating foundation matter. For a new company without comparable revenue and customer contracts, submit financial information and a business plan to obtain a company-specific credit decision. The advertised maximum helps select providers to approach.

I continued looking for providers that accept early-stage companies. NFS Capital’s official guidance explicitly considers startups and pre-revenue businesses through individual credit decisions. Depending on credit, asset loans may involve additional support such as cash accounts or securities, or structures involving warrants. Exxact’s sales FAQ also offers partner leasing and customized financing, with terms depending on location and credit history. Obtain the approved amount, down payment, funding price, guarantees, and draw conditions together to establish a complete proposal. This planning project has obtained no customer-specific quote or credit approval; the primary funding table includes zero equipment debt.

When financing quotes arrive, the repayment calendar needs a closer look: who guarantees the debt, when funds become available, what early repayment costs, and how the end-of-term buyout works. Principal, the quoted interest rate, and fees all go into the cash table. Only then can we see how much is saved today and how much is deferred.

Credit approval: The amount and conditions a lender agrees to offer after review.

Drawdown: Actually taking money from an approved borrowing arrangement.

Principal: The amount borrowed.

Interest rate: The rate used to calculate interest. Financing fees are counted separately.

Guarantee: A guarantor’s agreed responsibility to support repayment.

Collateral: Assets pledged to support repayment.

Warrant: A right to buy company shares on agreed terms.

Separate the payments through the contracts

A zero in the equipment-financing column does not put every dollar into the bank on day one. With two deliveries, purchase terms can still be negotiated around deposits, shipment or delivery, and acceptance. Colfax's public terms accept cash or approved credit and allow separately accepted purchase terms. NADDOD's Net 15, 30, and 45 terms require credit approval. Short payment terms and multiyear financing address different periods of capital use.

I separated the funding plan into two tasks: secure commitments covering irrevocable orders, then time cash arrival to each payment date. If the supplier accepts an acceptance balance, allocate that cash to acceptance; if it requires advance payment, move the funds earlier. Negotiate staged installation and staged payment separately. The monthly funding plan follows the obligations in the signed contract.

I first plotted the wages and electricity whose monthly rhythm was already known, to see how cash would leave the account. From Month 4, add the planned monthly rent of $2,227,500. Add procurement, other services, and data spending in their contractual months. This shows the calculated operating rhythm first, then provides places for incoming quotes.

The table uses the $165 per kW per month planning rate. Multiplying 13,500kW by that rate gives $2,227,500 per month. Electricity remains separate.

With outgoing payments dated, new funding can be calculated month by month: contract payments due + all current-month operations + the next two months' operating buffer − available bank cash. Call committed funds when this leaves a gap. Include Month 12 bonuses in the buffer ahead of time. Reconcile that buffer with the full risk allowance so the same contingency is not funded twice.

Staged funding requires an investment agreement covering committed amounts, notice periods, funding conditions, and due-date responsibilities. Cash can then reach the bank along the contract schedule while procurement remains backed by secured funding.

Cash flow: A record of when money enters and leaves the bank account.

Funding commitment: An investor's contractual obligation to contribute funds.

Cash received: Money already in the account and available to use.

Payment term: The period from the contractual starting event to the payment deadline. Net 15, 30, and 45 mean payment within 15, 30, or 45 days.

The Budget Has a Shape, and a Calendar

Returning to those original 660 servers, the list has become a system: 640 main machines, twenty spares, three halls on one campus totaling 13.5MW, twenty internal engineers and leads, and external on-site services. Servers, switches, first-year compensation, electricity, and the $20.0475 million rent estimate produce a current planning subtotal of about $350.57 million. Complete Year 1 spending adds optical parts O and storage and other services X. Apply the 10% risk allowance to that complete budget. Equipment debt remains zero.

The next step now had specific documents: suppliers receive the equipment and connection schedules; facility operators receive the three-hall capacity, fiber-route, and date requirements. Quotes and payment terms return to individual lines in the budget. The calendar still targets deliveries in Months 4 and 6, acceptance in Month 7, and 76 days of main training across Months 8–10. Every reply has a place to go.

The procurement plan now connects machine readiness with funding. But what does the 500B model running on those machines actually look like inside? Next, we set the server enclosure aside and step into the model.