[Training a 500B LLM from Scratch] 14/25: How Do We Install the Software on Our First Server?

In the previous article, we arranged a place for the training material and saved progress. Now we pick the first server with eight B200 GPUs, log in, and prepare it to start learning.

Our first-day goal is small: read a little material, train for a few steps on the GPUs, save progress, and continue from that saved point. Once this works, we can bring the same environment to the other machines. Eventually, 5,120 GPUs across 640 servers will train one 500B dense model together.

We will install the software in that order. At each stage, we will look at the problem it solves and how to check that it works.

First, let the machine recognize its eight GPUs

This plan uses an Ubuntu software combination supported by the server manufacturer, including the operating system, kernel, GPU driver, and services that manage connections inside the machine. Ubuntu manages the hardware and runs programs. The other components let it use the B200 GPUs and their connections correctly. We will fix this combination as the common starting point for all 640 machines.

Why start with a manufacturer-supported combination? We need to know which versions work together and how hardware problems will be supported. A custom Linux setup gives us more freedom to choose system components and update schedules. It also makes us responsible for checking how those changes work with the server hardware. For this fleet of matching machines, we accept the manufacturer's version constraints and focus our effort on training and fleet operations.

For example, NVIDIA provides DGX OS for DGX machines, and its official support table lists the requirements for DGX B200. For the eight-GPU B200 server we purchase, we will obtain the support list, installation media, and reinstallation instructions for that exact model. The specific versions are confirmed during procurement acceptance.

If a machine arrives with software installed, we check it against that list. If it needs reinstalling, we use the same list. Ubuntu's installation tool is called apt: point it at the manufacturer's or NVIDIA's official software repository and specify the packages and versions. It downloads and installs them and records their dependencies. Later, when we prepare 640 machines, these steps will become an automated installation recipe.

NVIDIA's driver deployment documentation also recommends this package-manager approach for managing driver branches, upgrades, and related software. NVIDIA also provides a standalone .run installer for environments that manage the installation process themselves. Our plan uses fixed repositories and packages so that installations can be repeated and their records checked.

After installing the driver and restarting, run nvidia-smi to take attendance: all eight B200 GPUs should be present and healthy. This tells us that the operating system can see them. The GPUs also exchange data through NVSwitch, so we must check the services that manage those connections. For B200, these are Fabric Manager and NVLink Subnet Manager. Install them according to the official documentation and the server's supported combination, then check the services and links.

The result we want from this stage is clear: the operating system recognizes all eight GPUs, and the connections between them are ready. Now we can turn to the training program.

Kernel: The core of an operating system that directly manages hardware resources and coordinates running programs.

GPU driver: Software that lets the operating system interact with the GPU, including managing memory, computation, and the device itself.

Package manager: A tool that downloads, installs, and updates software and records its dependencies. Ubuntu uses the apt package manager.

Fabric Manager: A service that manages GPU interconnections inside the server. On B200 systems, it works with NVLink Subnet Manager.

NVLink Subnet Manager: A service involved in managing the NVSwitch connections in a B200 server.

Give the training program its tools

Suppose we want the model to read a batch of tokens, make predictions, and adjust its parameters based on the prediction errors. A researcher writes this process in Python, but the GPU must eventually do the computation.

Our plan uses PyTorch for model computation and gradient calculation. PyTorch calls the CUDA runtime and computation libraries to run operations on the GPU. For example, matrix multiplication can use an implementation from cuBLAS. Start with this sequence:

Training program → PyTorch and GPU computation libraries → Driver → B200

We could install these tools one at a time and manage their version compatibility ourselves. We choose an official training container as our starting point, bringing a prepared set of tools to the server. A container is a separate software environment in which a program runs. An image is the template used to create that environment. The same image can be given to another machine.

PyTorch: A framework that helps us define models, perform computations, and calculate gradients.

CUDA: NVIDIA's GPU computing platform. Here, we mainly use its runtime and computation libraries to run calculations on the GPU.

Container image: A template containing a program and the software it depends on, used to create a container's runtime environment.

On our first reference server, Docker starts the container. It needs a companion tool called NVIDIA Container Toolkit to provide GPU access. Following the official installation guide, we first prepare the driver and Docker, then install the toolkit and configure GPU access. We then start a test container and check whether it can also see all eight GPUs.

This gives us a useful division of responsibility: the server's operating system and driver handle the hardware, while Python, PyTorch, and the computation libraries inside the container support the training program. When the training code changes, we mainly replace the container image. The system underneath continues to follow the supported combination we selected.

We also connect the training-material and checkpoint directories from Part 13 to the container so that the program can read and save files. The material and training progress remain in external storage when the container shuts down or is replaced.

Docker: A tool for building container images and starting containers. Our plan uses it to build images and troubleshoot on the reference server.

NVIDIA Container Toolkit: Tools that let containers use the server's NVIDIA GPU devices and driver capabilities.

Add our training code

PyTorch provides the basic computation. A 500B training run also needs implementations for building the model, saving progress, restoring training, and coordinating multiple GPUs. Our plan uses Megatron Core and the training code from the matching Megatron-LM version to organize this work, then adds the project's own code and configuration.

We start with the build recipe included in an official Megatron release. We keep its selected base container and dependencies and add our changes on top. The result is a project training image. This lets us use the official combination while keeping a clear record of what we changed.

During development, installing an extra Python package is convenient. Before running on 640 machines, we put that change into the build recipe and generate a new image. The project uses the upstream recipe's uv tool and lockfile to record dependency choices. For example, uv sync --locked prepares the environment from the lockfile. If the project declarations no longer match the lockfile, it reports an error so that we can resolve the version relationship first.

Installing components by hand offers more freedom. Building one image gives every machine the same artifact. We choose the latter and take responsibility for maintaining build records, version updates, and acceptance checks.

Megatron Core: A library of components for large-model training. The training program connects its model, parallel-computation, and checkpoint components into a training run.

Lockfile: A file recording the dependency versions selected for a project, used to recreate the same dependency combination.

Read more: the selected versions and build process

As of October 29, 2026, this plan retains the released Megatron Core 0.19.2 and uses the official container build file under the same tag. The file starts from nvcr.io/nvidia/pytorch:26.06-py3 and provides locked dependencies.

The 26.06 container release notes list Ubuntu 24.04, CUDA 13.3.0, and PyTorch 2.13.0a0+8145d630e8. Here, Ubuntu refers to the user space inside the image. The host operating system still follows the server manufacturer's support list. Megatron's recipe sets NCCL to 2.30.7-1+cuda13.3 and handles other dependencies.

Fetch the Megatron-LM repository, check out core_v0.19.2, and record the complete result of git rev-parse HEAD. Build the dependency environment using the main stage in that version's docker/Dockerfile.ci.dev, then add the training code from the same commit to the final project image. We can name the project image shan-training:mc0.19.2-cu13.3. NVIDIA's base-image name remains unchanged.

Before installation, check the container release notes for CUDA driver requirements, the server manufacturer's support range, and known issues. Save the base-image digest, build arguments, lockfiles, build logs, and final image digest. This is our selected baseline for acceptance testing. We will expand it to other servers after the single-server computation, communication, and recovery checks pass.

Read more: what do pip, venv, Conda, and source builds do?

pip installs Python packages. venv creates a separate project environment so that different projects keep their packages apart. Conda can also manage software through environment definitions. All of these tools can support development. Our production build follows the upstream uv and lockfile approach so that we maintain one dependency definition.

When adding a package, check whether it would also replace PyTorch or another core dependency in the official combination. The server's system installation process continues to manage the driver.

To change a training implementation or compile a custom GPU operator, fetch the selected source version, build it with matching CUDA Toolkit and compilation tools, and add the result to the project image. Record the installation steps in a Dockerfile and the full source commit so that another colleague can find the same code and build method.

Python virtual environment: An environment that keeps Python packages separate for one project.

Operator: A computation in a program, such as matrix multiplication. Its GPU implementation is commonly called a kernel.

CUDA Toolkit: NVIDIA's GPU development toolkit, including a compiler, runtime, and related tools and libraries.

Dockerfile: A file that records the steps for building a container image.

Commit ID: A unique identifier for a particular change in a code repository.

Run a little training, then check that it can resume

Once the software is installed, the most useful check is to make the pieces do real work together. We use a small model and a fixed sample of material to answer four questions in order.

First, can we use the GPUs? Confirm that all eight B200 GPUs are healthy on the server, then check again inside the container. The devices visible in both places should match.

Second, can we compute correctly? Run a calculation with a known answer on one GPU, then let the small model make predictions, calculate gradients, and update its parameters. Check that the values are valid, gradients are produced, and parameters change as expected. Include the BF16 computation path planned for training, along with the attention and fused operators it will use.

BF16: A floating-point format that uses two bytes per number. This plan uses it for part of the training computation while keeping higher-precision state where needed.

Third, can the eight GPUs exchange results? Give GPU 1 the number 1, GPU 2 the number 2, and so on through GPU 8. Ask them to sum the numbers and return the result to every GPU. The correct answer is 36.

This operation is called AllReduce and is performed by the NCCL communication library in the training environment. The training program supplies the data to exchange, and NCCL carries out the exchange. It supports communication both inside a machine and across machines, as described in the official NCCL documentation. After the small test passes, test several larger data sizes and save the timings and logs for comparison when accepting other machines.

AllReduce: Combine values from the participating devices, then give the combined result to each device. Our example uses summation.

NCCL: NVIDIA's GPU communication library, which performs operations such as reduction, broadcast, and gather.

Fourth, can training continue after it stops? Read a token file in the actual training format from the storage location arranged in Part 13. Plan a short run of 100 steps and save a checkpoint at step 50. Then start a new process and continue from that checkpoint to step 100.

Starting a new process confirms that recovery depends on the saved state. The new process must restore model parameters, the optimizer, the step count, random state, and the data-reading position. The material then continues from the correct place, and the learning rate follows its original schedule. The PyTorch saving and loading tutorial explains the difference between model weights and a state from which training can resume.

We also run 100 steps continuously as a comparison. Both runs use the same environment, material order, and computation settings. Compare the key states and numerical error, with the allowed tolerance set before the test. Floating-point computation and some GPU operators affect bit-for-bit agreement. The PyTorch reproducibility notes explain the relevant conditions.

Only after all four checks pass can we treat this machine as a reference. These checks cover devices, computation, communication, and training recovery. The actual throughput of the 500B model and the write speed of a complete 7TB checkpoint belong in the later tests at scale.

Expand to 640 servers: choose how to manage them, then copy the environment

On the first machine, we can start a program directly. With 640 machines, we first need to decide which machines belong to which task, when tasks start, and how failures are handled.

This plan uses Slurm to manage the main training servers. When submitting training, we first request the entire group of nodes required. Once resources are allocated, we start and monitor the training processes on those nodes. Small tests also request resources through the queue, and usage is recorded by account. These responsibilities match resource allocation, parallel-job management, and queuing in the official Slurm overview.

We choose Slurm because the main job of these 640 machines is long-running, synchronous training of a 500B model: many GPUs compute together and continually exchange results. Slurm organizes resources around parallel jobs of this kind, which fits our project. Operations handover and training will cover the control service, node services, accounting, and failure handling.

Kubernetes is another option. It can also organize distributed training and is useful when training, online inference, and applications from multiple teams share one platform. An organization with an established Kubernetes team can reuse that capability. Our 640 machines are dedicated to main training, so the benefit of mixing in online services is smaller. We therefore select Slurm and concentrate on maintaining that route. Actual training speed will still be measured through computation and communication tests.

Slurm: A scheduler that allocates compute resources to cluster tasks and handles queuing, startup, and monitoring.

Kubernetes: A platform for managing containerized workloads, with capabilities that can be combined for services, batch jobs, and distributed training.

Slurm decides which machines run the job. We also need to start the containers themselves. This plan uses Pyxis to connect container tasks to Slurm, with Enroot providing the runtime environment. After a user submits a job, it can run the project image on the allocated machines.

We used Docker earlier to build the image and troubleshoot on the reference machine. Before rolling out, we repeat the same acceptance checks through Slurm, Pyxis, and Enroot, covering GPUs, file directories, communication, and recovery. We pin the source image by digest, record a separate checksum for the imported artifact, and distribute it to the nodes.

Pyxis: A plugin that connects container tasks to Slurm.

Enroot: A tool that imports container images and runs them as user-space environments.

Image digest: An identifier derived from image content, used to confirm that we retrieve the same image.

Read more: how do other platforms work, and what would we maintain?

Kubernetes can use Kueue for queues and quotas and Kubeflow Trainer to organize distributed training. GPU Operator can manage GPU drivers, Container Toolkit, device plugins, and monitoring components. Administrators also need to coordinate component upgrades and the permissions, quotas, and isolation of a shared cluster.

The benefit is reuse of a common application platform. Its maintenance scope also includes the selected queue, training controller, networking, and device integrations. Our plan chooses Slurm and manages host software through a common installation recipe.

Apptainer is another container tool with GPU support, suitable for computing centers that adopt it as their standard runtime. Our production environment uses Pyxis and Enroot so that we maintain one launch method. The official Pyxis documentation requires it to match the deployed Slurm version. That relationship must also appear in the version record.

Kueue: A queuing and quota-management system on Kubernetes that decides when a task is admitted to run.

Kubeflow Trainer: A component that organizes distributed model training on Kubernetes.

GPU Operator: A tool that automatically manages GPU-related software components on Kubernetes.

Install every machine from the same recipe

Before the scheduler can assign work, each machine must be prepared in the same way. We need to copy two things: the server's system environment and the project training image.

The container image comes from the team's registry. An automated installation recipe builds the system environment. It follows the manufacturer-supported Ubuntu combination selected earlier. This plan uses PXE, Ubuntu autoinstall, cloud-init, and Ansible in sequence, with a concrete result at each step:

Ubuntu autoinstall, cloud-init, and Ansible provide official instructions for their parts. We keep one common recipe and supply each machine's address and identity separately. We also archive the validated system images and packages for reinstallation and rollback.

An integrated platform such as NVIDIA Base Command Manager offers another approach. It manages nodes, system images, distribution, and updates in one system. This reduces the work of connecting separate tools, while adding that platform and its integrations to the maintenance scope.

Our plan selects the separate tool combination above. Infrastructure staff within the 20-person team work with the systems integrator to deliver and maintain the installation recipe, reviewing system versions, configuration, and acceptance steps together. The tradeoff is that we must integrate installation orchestration, log collection, and version records. We therefore include those in delivery acceptance. Each machine follows one authoritative recipe, and problems can be traced through the same records.

PXE: A way for a server to obtain a startup program over the network.

cloud-init: A tool that initializes a system according to configuration.

Ansible: A tool for installing software, configuring services, and managing servers in batches from configuration.

Follow the same path when upgrading

After the first reference server passes, expand to a small group and then roll out in batches. Check devices, computation, and communication inside each machine. A failed node joins the available fleet after it is repaired and passes again. Communication between machines and overall performance are covered by later cluster acceptance tests.

Upgrades follow the same process. When training code or dependencies change, build a new image and run the short training and recovery tests on the reference server. After they pass, use it on a small group, then switch the remaining machines in batches. Keep the previous image in the registry so that the tested version is available for rollback.

Remember the image digest. Docker's documentation explains that an image tag can point to updated content. A tag helps people recognize an image; a digest pins its specific content. We create a new artifact for a new environment, record its new digest, and use it after validation. This is the immutable-image approach.

For an operating-system kernel or driver update, schedule a maintenance window, keep restorable system images and packages, and repeat acceptance checks after restarting. When rolling back the training environment, also confirm that the older version can read the checkpoint intended for recovery.

The handover to a colleague should be a process they can repeat: install the system from the records, retrieve the specified image, connect the material and checkpoint directories, run the same tests, and judge whether they pass.

Read more: which records belong in the environment handover?

Record	Purpose
Hardware, firmware, operating-system, kernel, and driver versions	Identify the machine and its underlying software conditions
Management services, container tools, packages, and configuration	Reinstall the same system and production runtime
Training-code commit, dependency lockfile, and build recipe	Retrieve the program and build method used at the time
Source-image digest and imported-artifact checksum	Retrieve the same training environment
Training configuration, tokenizer, sample data, and checksums	Repeat the same training sample
Launch commands, directory locations, pass criteria, and test logs	Let a colleague repeat the procedure and judge the result
Previous environment and checkpoint-compatibility records	Restore training if an update causes a problem

These results tell us whether an installation approach fits large-scale training in 2026: it can be repeated, its software contents can be identified, acceptance can be automated, and an update can be rolled back to a known working state.

We have now selected the installation and runtime path from the first machine to all 640. Next, we return to training itself: a 500B model does not fit on one GPU, so how do we divide it across many GPUs and make those pieces train together?